# How to Set Up ai-memory with Anthropic, OpenAI, and Gemini LLM Providers

> Easily set up ai-memory with Anthropic, OpenAI, or Gemini LLM providers. Configure the AI_MEMORY_LLM_PROVIDER environment variable and export your API key for seamless integration.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Configure ai-memory by setting the `AI_MEMORY_LLM_PROVIDER` environment variable to `anthropic`, `openai`, or `gemini`, export the corresponding API key, and the `ProviderFactory` in [`crates/ai-memory-llm/src/factory.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/factory.rs) automatically instantiates the correct client with typed credentials.**

The ai-memory project (akitaonrails/ai-memory) abstracts LLM interactions through the `ai-memory-llm` crate, allowing you to set up ai-memory with different LLM providers using a unified configuration surface. The system reads `AI_MEMORY_*` environment variables at startup, validates them in [`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs), and routes requests to provider-specific implementations that handle authentication, serialization, and error handling.

## How Provider Selection Works

The LLM subsystem uses a factory pattern to decouple provider-specific logic from the core application. When the server starts, `Config::load` (defined in [`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs)) ingests all environment variables into a typed struct. If any LLM-dependent feature—such as memory consolidation or reranking—is enabled, the system checks for `AI_MEMORY_LLM_PROVIDER`.

`ProviderFactory::build` (in [`crates/ai-memory-llm/src/factory.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/factory.rs)) matches the provider name against the `ProviderChoice` enum variants (`Anthropic`, `AnthropicOAuth`, `OpenAI`, `Gemini`). It then retrieves the required secret from the `Config` struct and instantiates the concrete implementation: `AnthropicProvider`, `OpenAiProvider`, or `GeminiProvider`. This provider is wrapped in a `LlmClient` and injected into the consolidation pipeline (`ai-memory-consolidate`) and the reranker module.

## Provider-Specific Configuration

Each provider requires a distinct environment variable for authentication and offers a default model when `AI_MEMORY_LLM_MODEL` is omitted.

### Anthropic

**Required variable:** `AI_MEMORY_ANTHROPIC_API_KEY`

**Authentication:** Header `x-api-key` sent with every request

**Default models:** `claude-haiku-4.5` (standard) or `claude-sonnet-4.5` (higher quality tier)

```bash
export AI_MEMORY_LLM_PROVIDER=anthropic
export AI_MEMORY_ANTHROPIC_API_KEY="sk-ant-xxxxxxxxxxxxxxxxxxxx"
export AI_MEMORY_LLM_MODEL=claude-sonnet-4-5  #optional

```

For Claude subscription accounts, use the `anthropic-oauth` provider instead:

```bash
export AI_MEMORY_LLM_PROVIDER=anthropic-oauth
export AI_MEMORY_ANTHROPIC_OAUTH_TOKEN="sk-ant-..."

```

### OpenAI

**Required variable:** `AI_MEMORY_OPENAI_API_KEY`

**Authentication:** Header `Authorization: Bearer <token>`

**Default model:** `gpt-4o-mini`

```bash
export AI_MEMORY_LLM_PROVIDER=openai
export AI_MEMORY_OPENAI_API_KEY="sk-xxxxxxxxxxxxxxxxxxxx"

```

### Gemini

**Required variable:** `AI_MEMORY_GEMINI_API_KEY`

**Authentication:** Header `Authorization: Bearer <token>`

**Default model:** `gemini-1.5-flash`

```bash
export AI_MEMORY_LLM_PROVIDER=gemini
export AI_MEMORY_GEMINI_API_KEY="xxxxxxxxxxxxxxxxxxxx"

```

## Optional Global Settings

All providers share these additional configuration variables:

- `AI_MEMORY_LLM_MODEL` – overrides the builtin default model
- `AI_MEMORY_LLM_TIMEOUT_SECS` – request timeout (default varies)
- `AI_MEMORY_LLM_COMPAT_STRICT` – when `true` (default), sends `response_format={type:"json_schema",strict:true}` to enforce valid JSON output; set to `false` to disable strict schema validation

## Quick Start Guide

Follow these steps to activate LLM features in ai-memory.

1. **Obtain credentials**
   - Anthropic: Generate an API key at https://console.anthropic.com/
   - OpenAI: Create a key at https://platform.openai.com/account/api-keys
   - Gemini: Enable the Generative Language API in Google Cloud and create an API key

2. **Configure environment variables**

Create a `.env` file or export directly:

```bash
export AI_MEMORY_LLM_PROVIDER=anthropic
export AI_MEMORY_ANTHROPIC_API_KEY="sk-ant-..."
export AI_MEMORY_LLM_COMPAT_STRICT=true

```

3. **Start the server**

```bash
ai-memory serve

# Listens on 127.0.0.1:49374 by default

```

4. **Verify connectivity**

The CLI includes a test command that exercises the selected provider:

```bash
ai-memory llm-test --provider anthropic

# Expected output:

# → Provider: Anthropic

# → Model: claude-sonnet-4.5

# → Prompt succeeded, response size: 312 tokens

```

## Programmatic Usage

You can instantiate providers directly in Rust code using the factory:

```rust
use ai_memory_llm::factory::ProviderFactory;
use ai_memory_llm::ProviderChoice;

let config = ai_memory_cli::config::Config::load().unwrap();
let provider = ProviderFactory::build(&config).unwrap();

let response = provider
    .complete_structured(
        "You are a helpful assistant. Summarize the following text:",
        "The quick brown fox jumps over the lazy dog.",
        None,  // no custom system prompt
        None,  // no tool usage
        &config,
    )
    .await
    .expect("LLM call failed");

println!("LLM answer: {}", response.text);

```

Each provider implementation handles request construction internally:
- **Anthropic** ([`crates/ai-memory-llm/src/anthropic.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/anthropic.rs)): POSTs to `https://api.anthropic.com/v1/messages` with `x-api-key`
- **OpenAI** ([`crates/ai-memory-llm/src/openai.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/openai.rs)): POSTs to OpenAI-compatible endpoints with `Authorization: Bearer`
- **Gemini** ([`crates/ai-memory-llm/src/gemini.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/gemini.rs)): POSTs to `https://generativelanguage.googleapis.com/v1/models/...:generateContent` with Bearer authentication

Responses are parsed using a tolerant parser first; if `AI_MEMORY_LLM_COMPAT_STRICT` is enabled and the server returns a JSON schema, the response is validated immediately against the schema.

## Summary

- **Factory pattern**: `ProviderFactory::build` in [`crates/ai-memory-llm/src/factory.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/factory.rs) instantiates the correct provider based on `AI_MEMORY_LLM_PROVIDER`
- **Three supported backends**: Anthropic (API key or OAuth), OpenAI, and Gemini—each with distinct authentication headers and default models
- **Environment-driven**: All credentials and settings load via `Config::load` in [`crates/ai-memory-cli/src/config.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-cli/src/config.rs) at server startup
- **Strict mode**: JSON schema validation is enabled by default via `AI_MEMORY_LLM_COMPAT_STRICT=true` for structured outputs
- **Testing**: Use `ai-memory llm-test` to verify provider connectivity before running consolidation jobs

## Frequently Asked Questions

### What happens if I don't set `AI_MEMORY_LLM_PROVIDER`?

If the environment variable is empty or unset, ai-memory disables LLM-dependent features (consolidation, auto-improve, reranking) and operates in local-only mode. The server starts normally but skips initializing the `LlmClient`.

### Can I use a custom model not listed in the defaults?

Yes. Set `AI_MEMORY_LLM_MODEL` to any valid model identifier supported by your provider. The factory passes this string directly to the provider implementation, which inserts it into the API request payload. For example, `export AI_MEMORY_LLM_MODEL=gpt-4o` overrides the default `gpt-4o-mini` for OpenAI.

### How do I disable strict JSON schema validation?

Set `AI_MEMORY_LLM_COMPAT_STRICT=false`. By default, the system sends `response_format={type:"json_schema",strict:true}` to OpenAI-compatible providers to ensure valid JSON. Disabling this allows the tolerant parser to handle malformed or free-text responses without throwing validation errors.

### What's the difference between `anthropic` and `anthropic-oauth` providers?

The `anthropic` provider uses `AI_MEMORY_ANTHROPIC_API_KEY` and sends the `x-api-key` header for standard API access. The `anthropic-oauth` variant uses `AI_MEMORY_ANTHROPIC_OAUTH_TOKEN` for Claude subscription accounts that authenticate via OAuth tokens rather than standard API keys. Both are implemented in [`crates/ai-memory-llm/src/anthropic.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/anthropic.rs) but use different credential fields from the `Config` struct.