How to Set Up ai-memory with Anthropic, OpenAI, and Gemini LLM Providers

Configure ai-memory by setting the AI_MEMORY_LLM_PROVIDER environment variable to anthropic, openai, or gemini, export the corresponding API key, and the ProviderFactory in crates/ai-memory-llm/src/factory.rs automatically instantiates the correct client with typed credentials.

The ai-memory project (akitaonrails/ai-memory) abstracts LLM interactions through the ai-memory-llm crate, allowing you to set up ai-memory with different LLM providers using a unified configuration surface. The system reads AI_MEMORY_* environment variables at startup, validates them in crates/ai-memory-cli/src/config.rs, and routes requests to provider-specific implementations that handle authentication, serialization, and error handling.

How Provider Selection Works

The LLM subsystem uses a factory pattern to decouple provider-specific logic from the core application. When the server starts, Config::load (defined in crates/ai-memory-cli/src/config.rs) ingests all environment variables into a typed struct. If any LLM-dependent feature—such as memory consolidation or reranking—is enabled, the system checks for AI_MEMORY_LLM_PROVIDER.

ProviderFactory::build (in crates/ai-memory-llm/src/factory.rs) matches the provider name against the ProviderChoice enum variants (Anthropic, AnthropicOAuth, OpenAI, Gemini). It then retrieves the required secret from the Config struct and instantiates the concrete implementation: AnthropicProvider, OpenAiProvider, or GeminiProvider. This provider is wrapped in a LlmClient and injected into the consolidation pipeline (ai-memory-consolidate) and the reranker module.

Provider-Specific Configuration

Each provider requires a distinct environment variable for authentication and offers a default model when AI_MEMORY_LLM_MODEL is omitted.

Anthropic

Required variable: AI_MEMORY_ANTHROPIC_API_KEY

Authentication: Header x-api-key sent with every request

Default models: claude-haiku-4.5 (standard) or claude-sonnet-4.5 (higher quality tier)

export AI_MEMORY_LLM_PROVIDER=anthropic
export AI_MEMORY_ANTHROPIC_API_KEY="sk-ant-xxxxxxxxxxxxxxxxxxxx"
export AI_MEMORY_LLM_MODEL=claude-sonnet-4-5  #optional

For Claude subscription accounts, use the anthropic-oauth provider instead:

export AI_MEMORY_LLM_PROVIDER=anthropic-oauth
export AI_MEMORY_ANTHROPIC_OAUTH_TOKEN="sk-ant-..."

OpenAI

Required variable: AI_MEMORY_OPENAI_API_KEY

Authentication: Header Authorization: Bearer <token>

Default model: gpt-4o-mini

export AI_MEMORY_LLM_PROVIDER=openai
export AI_MEMORY_OPENAI_API_KEY="sk-xxxxxxxxxxxxxxxxxxxx"

Gemini

Required variable: AI_MEMORY_GEMINI_API_KEY

Authentication: Header Authorization: Bearer <token>

Default model: gemini-1.5-flash

export AI_MEMORY_LLM_PROVIDER=gemini
export AI_MEMORY_GEMINI_API_KEY="xxxxxxxxxxxxxxxxxxxx"

Optional Global Settings

All providers share these additional configuration variables:

• AI_MEMORY_LLM_MODEL – overrides the builtin default model
• AI_MEMORY_LLM_TIMEOUT_SECS – request timeout (default varies)
• AI_MEMORY_LLM_COMPAT_STRICT – when true (default), sends response_format={type:"json_schema",strict:true} to enforce valid JSON output; set to false to disable strict schema validation

Quick Start Guide

Follow these steps to activate LLM features in ai-memory.

1. Obtain credentials

   • Anthropic: Generate an API key at https://console.anthropic.com/
   • OpenAI: Create a key at https://platform.openai.com/account/api-keys
   • Gemini: Enable the Generative Language API in Google Cloud and create an API key

2. Configure environment variables

Create a .env file or export directly:

export AI_MEMORY_LLM_PROVIDER=anthropic
export AI_MEMORY_ANTHROPIC_API_KEY="sk-ant-..."
export AI_MEMORY_LLM_COMPAT_STRICT=true

3. Start the server

ai-memory serve

# Listens on 127.0.0.1:49374 by default

4. Verify connectivity

The CLI includes a test command that exercises the selected provider:

ai-memory llm-test --provider anthropic

# Expected output:

# → Provider: Anthropic

# → Model: claude-sonnet-4.5

# → Prompt succeeded, response size: 312 tokens

Programmatic Usage

You can instantiate providers directly in Rust code using the factory:

use ai_memory_llm::factory::ProviderFactory;
use ai_memory_llm::ProviderChoice;

let config = ai_memory_cli::config::Config::load().unwrap();
let provider = ProviderFactory::build(&config).unwrap();

let response = provider
    .complete_structured(
        "You are a helpful assistant. Summarize the following text:",
        "The quick brown fox jumps over the lazy dog.",
        None,  // no custom system prompt
        None,  // no tool usage
        &config,
    )
    .await
    .expect("LLM call failed");

println!("LLM answer: {}", response.text);

Each provider implementation handles request construction internally:

• Anthropic (crates/ai-memory-llm/src/anthropic.rs): POSTs to https://api.anthropic.com/v1/messages with x-api-key
• OpenAI (crates/ai-memory-llm/src/openai.rs): POSTs to OpenAI-compatible endpoints with Authorization: Bearer
• Gemini (crates/ai-memory-llm/src/gemini.rs): POSTs to https://generativelanguage.googleapis.com/v1/models/...:generateContent with Bearer authentication

Responses are parsed using a tolerant parser first; if AI_MEMORY_LLM_COMPAT_STRICT is enabled and the server returns a JSON schema, the response is validated immediately against the schema.

Summary

• Factory pattern: ProviderFactory::build in crates/ai-memory-llm/src/factory.rs instantiates the correct provider based on AI_MEMORY_LLM_PROVIDER
• Three supported backends: Anthropic (API key or OAuth), OpenAI, and Gemini—each with distinct authentication headers and default models
• Environment-driven: All credentials and settings load via Config::load in crates/ai-memory-cli/src/config.rs at server startup
• Strict mode: JSON schema validation is enabled by default via AI_MEMORY_LLM_COMPAT_STRICT=true for structured outputs
• Testing: Use ai-memory llm-test to verify provider connectivity before running consolidation jobs

Frequently Asked Questions

What happens if I don't set AI_MEMORY_LLM_PROVIDER?

If the environment variable is empty or unset, ai-memory disables LLM-dependent features (consolidation, auto-improve, reranking) and operates in local-only mode. The server starts normally but skips initializing the LlmClient.

Can I use a custom model not listed in the defaults?

Yes. Set AI_MEMORY_LLM_MODEL to any valid model identifier supported by your provider. The factory passes this string directly to the provider implementation, which inserts it into the API request payload. For example, export AI_MEMORY_LLM_MODEL=gpt-4o overrides the default gpt-4o-mini for OpenAI.

How do I disable strict JSON schema validation?

Set AI_MEMORY_LLM_COMPAT_STRICT=false. By default, the system sends response_format={type:"json_schema",strict:true} to OpenAI-compatible providers to ensure valid JSON. Disabling this allows the tolerant parser to handle malformed or free-text responses without throwing validation errors.

What's the difference between anthropic and anthropic-oauth providers?

The anthropic provider uses AI_MEMORY_ANTHROPIC_API_KEY and sends the x-api-key header for standard API access. The anthropic-oauth variant uses AI_MEMORY_ANTHROPIC_OAUTH_TOKEN for Claude subscription accounts that authenticate via OAuth tokens rather than standard API keys. Both are implemented in crates/ai-memory-llm/src/anthropic.rs but use different credential fields from the Config struct.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}