How to Configure OpenAI, Voyage, and Ollama Embedding Providers for Vector Search in ai-memory

Configure embedding providers in ai-memory by setting the AI_MEMORY_EMBEDDER environment variable to openai, voyage, or ollama, then provide the corresponding API keys or base URLs and restart the service.

The ai-memory project implements a pluggable embedding architecture through the ai-memory-llm crate, allowing you to generate vector embeddings using cloud-based APIs or local inference. The configuration is loaded at startup via Figment from environment variables, enabling seamless switching between OpenAI, Voyage AI, and Ollama without code changes.

Selecting an Embedding Provider

The system selects embedders using a factory pattern defined in crates/ai-memory-llm/src/factory.rs. Set the AI_MEMORY_EMBEDDER variable to one of the following supported values:

• openai – Cloud-based embeddings via OpenAI API
• voyage – Cloud-based embeddings via Voyage AI API
• ollama – Local embeddings via Ollama daemon

Each provider requires specific authentication variables processed by the configuration loader in crates/ai-memory-llm/src/lib.rs.

OpenAI Configuration

Set either OPENAI_API_KEY or AI_MEMORY_OPENAI_API_KEY with your OpenAI API credentials. The provider implementation handles authentication headers in crates/ai-memory-llm/src/provider.rs.

AI_MEMORY_EMBEDDER=openai
AI_MEMORY_EMBEDDER_MODEL=text-embedding-3-large
OPENAI_API_KEY=sk-XXXXXXXXXXXXXXXXXXXXXXXX

Voyage Configuration

Provide your Voyage API key via the VOYAGE_API_KEY environment variable.

AI_MEMORY_EMBEDDER=voyage
AI_MEMORY_EMBEDDER_MODEL=voyage-2
VOYAGE_API_KEY=your-voyage-key-here

Ollama Configuration

Specify the base URL for your local Ollama instance. If omitted, it defaults to http://localhost:11434.

AI_MEMORY_EMBEDDER=ollama
AI_MEMORY_EMBEDDER_MODEL=llama2
OLLAMA_BASE_URL=http://127.0.0.1:11434

Specifying Embedding Models

Use AI_MEMORY_EMBEDDER_MODEL to define which model variant the provider should use for vector generation. This parameter is passed directly to the provider's embedding endpoint.

• OpenAI: text-embedding-3-small, text-embedding-3-large, or text-embedding-ada-002
• Voyage: voyage-2, voyage-large-2, or other Voyage model codes
• Ollama: Any installed model name (e.g., llama2, mistral, nomic-embed-text)

How the Embedding Pipeline Works

When ai-memory starts, the create_embedder function in crates/ai-memory-llm/src/factory.rs instantiates an object implementing the Embedder trait defined in crates/ai-memory-llm/src/embedding.rs. This embedder is injected into the consolidation pipeline.

During wiki page processing, the embed::run function in crates/ai-memory-consolidate/src/embed.rs calls embedder.embed(text) for each document. The resulting vectors are stored in SQLite alongside page content, enabling similarity search through the /api/v1/search API endpoint.

Testing Your Configuration

Start the binary to verify the embedder initializes correctly:

cargo run --bin ai-memory

Perform a vector search via the HTTP API:

curl -X POST http://127.0.0.1:49374/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query":"how to configure embeddings","top_k":5}'

The request body is converted to an embedding using your configured provider, and the system returns the nearest stored vectors.

Programmatic Usage

For custom Rust implementations, use the factory function directly as shown in crates/ai-memory-llm/tests/openai_compat_embedder.rs:

use ai_memory_llm::factory::create_embedder;
use ai_memory_llm::embedding::Embedder;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let embedder = create_embedder().await?;
    let vector = embedder.embed("example text for embedding").await?;
    println!("Generated {}-dimensional vector", vector.len());
    Ok(())
}

Summary

• Set AI_MEMORY_EMBEDDER to openai, voyage, or ollama to select your provider architecture
• Provide authentication via OPENAI_API_KEY (or AI_MEMORY_OPENAI_API_KEY), VOYAGE_API_KEY, or OLLAMA_BASE_URL
• Define the model dimensionality and behavior with AI_MEMORY_EMBEDDER_MODEL
• The factory in crates/ai-memory-llm/src/factory.rs constructs the appropriate embedder implementing the trait from crates/ai-memory-llm/src/embedding.rs
• Restart the service after changing providers or models to reinitialize the Figment configuration loader

Frequently Asked Questions

Can I switch embedding providers without rebuilding ai-memory?

Yes. The embedder is selected at runtime through environment variables processed by Figment in crates/ai-memory-llm/src/lib.rs. Update your .env file or export the new variables, then restart the binary. The factory will instantiate the new provider implementation without requiring recompilation.

Does ai-memory support multiple embedding providers simultaneously?

No. The system initializes a single global embedder instance via factory::create_embedder during startup. All vector generation operations in the consolidation pipeline use this single configured provider. To use a different provider, you must restart the service with new environment variables.

What happens if I change the embedding model after data is already stored?

Changing AI_MEMORY_EMBEDDER_MODEL creates vectors incompatible with existing embeddings in the database, as different models produce different dimensional spaces. You must regenerate embeddings for existing wiki pages when switching models. The system does not automatically re-embed existing content upon configuration changes.

Is there a fallback if the Ollama server is unreachable?

The system does not provide automatic fallback between providers. If AI_MEMORY_EMBEDDER=ollama and the OLLAMA_BASE_URL endpoint is unavailable, the embedder will return connection errors when the consolidation pipeline attempts to generate vectors. Ensure your Ollama daemon is running before starting ai-memory when using local embedding models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}