# How ai-memory Manages Multiple LLM Providers with Automatic Fallback Chains

> Discover how ai-memory seamlessly manages multiple LLM providers with automatic fallback chains. Learn about its circuit-breaker logic for resilient performance.

- Repository: [Fabio Akita/ai-memory](https://github.com/akitaonrails/ai-memory)
- Tags: how-to-guide
- Published: 2026-09-10

---

**ai-memory unifies diverse LLM backends behind a single `FallbackLlmProvider` wrapper that implements the `LlmProvider` trait, automatically cycling through configured providers when transient errors occur while using circuit-breaker logic to skip temporarily unhealthy endpoints.**

The **ai-memory** project by akitaonrails delivers a resilient abstraction layer for large language model integration. Instead of locking users to a single vendor, the system supports configuring multiple LLM providers in a prioritized chain, ensuring your application remains operational even when individual services experience downtime. This architecture centers on the `LlmProvider` trait defined in [`crates/ai-memory-llm/src/provider.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/provider.rs), which standardizes interactions across diverse backends including Anthropic, Google Gemini, and OpenAI-compatible endpoints.

## The LlmProvider Trait Abstraction

All LLM implementations in ai-memory conform to the `LlmProvider` trait defined in [`crates/ai-memory-llm/src/provider.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/provider.rs). This interface abstracts provider-specific details, exposing uniform methods like `complete` and `complete_with_operation_id` that the rest of the application uses for tasks such as consolidation, linting, and query reranking.

By coding against this trait rather than concrete implementations, ai-memory achieves polymorphic behavior where the consumer cannot distinguish between a single provider and an entire fallback chain. The trait contract requires implementations to return standardized `LlmError` types, which include the critical `is_transient()` method used by the fallback logic to distinguish retryable failures from configuration errors.

## Configuring the Provider Chain

The CLI reads the `llm_provider_chain` configuration key at startup to construct an ordered list of providers. Located in [`crates/ai-memory-llm/src/factory.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/factory.rs), the `build_provider` function parses this configuration and instantiates concrete provider implementations—whether Anthropic's Claude, Google's Gemini, or self-hosted OpenAI-compatible endpoints—wrapping them in `Candidate` structs that maintain health state.

Environment variables control this behavior at runtime:

```bash
export AI_MEMORY_LLM_PROVIDER_CHAIN="anthropic,gemini"
export AI_MEMORY_LLM_MODEL="claude-haiku-4-5,gemini-3.5-flash"
export AI_MEMORY_LLM_COOLDOWN_SECS=30

```

Each entry in the provider chain maps to a specific model configuration, allowing fine-grained control over which model variant serves as primary, secondary, or tertiary options.

## FallbackLlmProvider Implementation

The `FallbackLlmProvider` struct in [`crates/ai-memory-llm/src/fallback.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/fallback.rs) orchestrates the multi-provider logic. It maintains an internal `Vec<Candidate>` and implements the same `LlmProvider` trait as individual backends, making it a drop-in replacement throughout the codebase.

### Ordered Attempt Logic

When processing a request, the fallback provider iterates sequentially over `self.entries`. For each request method like `complete` or `complete_with_operation_id`, the implementation attempts the operation on the first candidate. Only if that attempt returns a transient error does the loop advance to the next candidate in the chain.

### Transient vs Deterministic Error Handling

Error handling distinguishes between recoverable and non-recoverable failures through the `LlmError::is_transient()` method. Transient errors—such as network timeouts or 5xx server errors—trigger fallback to the next provider. Deterministic errors—such as 400 bad requests, 401 authentication failures, or schema violations—cause immediate short-circuiting of the entire chain.

As implemented in [`fallback.rs`](https://github.com/akitaonrails/ai-memory/blob/main/fallback.rs) lines 93-95:

```rust
if !transient {
    return Err(err);
}

```

This prevents wasting API calls on requests that will fail identically across all providers.

### Circuit Breaker Pattern

To prevent hammering unhealthy providers, ai-memory implements a circuit breaker with a default cooldown of 30 seconds (`CIRCUIT_COOLDOWN`). When a candidate records a transient error via `record_error`, the code sets `circuit_open_until` to 30 seconds in the future. The `circuit_open` method checks this timestamp and skips candidates whose circuits remain open.

Successful calls immediately clear the circuit through `record_success`, which resets `circuit_open_until` to `None`. This allows fast recovery detection when a provider returns to health.

### Health Monitoring

Each candidate maintains detailed health metrics accessible via `candidate_health()`. Defined in [`crates/ai-memory-llm/src/health.rs`](https://github.com/akitaonrails/ai-memory/blob/main/crates/ai-memory-llm/src/health.rs), the `CandidateHealth` struct exposes:

- `last_success_at`: Timestamp of the last successful request
- `last_error_class`: Classification of the most recent failure  
- `last_error_status`: HTTP status code of the last error
- `circuit_open_until`: Optional timestamp indicating when the circuit closes

Administrators can query these metrics to observe provider reliability and tune the fallback chain configuration.

## Practical Usage Examples

Building a fallback chain programmatically follows this pattern:

```rust
use ai_memory_llm::fallback::{FallbackLlmProvider, Candidate};
use std::sync::Arc;

let providers = vec![
    Candidate::new(
        "anthropic",
        "claude-haiku-4-5",
        Arc::new(ai_memory_llm::anthropic::AnthropicProvider::new()),
    ),
    Candidate::new(
        "gemini",
        "gemini-3.5-flash",
        Arc::new(ai_memory_llm::gemini::GeminiProvider::new()),
    ),
];

let llm = FallbackLlmProvider::new(providers);

```

Once constructed, usage is identical to single providers:

```rust
let request = ChatRequest::user_prompt("Explain Rust ownership")
    .with_max_tokens(500);
let response = llm.complete(request).await?;
println!("Response: {}", response.text);

```

When all candidates exhaust, the system returns `LlmError::AllCandidatesFailed`, aggregating attempt counts and redacted error summaries for debugging without exposing sensitive tokens.

## Summary

- **ai-memory** abstracts LLM interactions through the `LlmProvider` trait in [`provider.rs`](https://github.com/akitaonrails/ai-memory/blob/main/provider.rs), enabling polymorphic provider handling.
- The `FallbackLlmProvider` in [`fallback.rs`](https://github.com/akitaonrails/ai-memory/blob/main/fallback.rs) wraps multiple candidates in a prioritized chain with automatic failover.
- Only **transient errors** trigger fallback; deterministic errors like authentication failures short-circuit immediately.
- A **30-second circuit breaker** prevents retry storms against unhealthy providers, resetting automatically on success.
- Comprehensive **health metrics** per candidate support operational monitoring and debugging.
- Configuration through environment variables like `AI_MEMORY_LLM_PROVIDER_CHAIN` enables runtime customization without code changes.

## Frequently Asked Questions

### How does ai-memory decide when to switch to the next LLM provider?

The `FallbackLlmProvider` iterates through its internal `Vec<Candidate>` sequentially. When a provider returns an error, the code checks `err.is_transient()`. If the error is transient (e.g., network timeout, 5xx status), the loop continues to the next candidate. Non-transient errors abort the chain immediately.

### What is the default cooldown period for failed providers?

The default `CIRCUIT_COOLDOWN` is **30 seconds**, configurable via the `AI_MEMORY_LLM_COOLDOWN_SECS` environment variable. When a provider fails with a transient error, the circuit breaker marks it as unhealthy for this duration before allowing retry attempts.

### Can I mix commercial and self-hosted LLM providers in the same chain?

Yes. The factory pattern in [`factory.rs`](https://github.com/akitaonrails/ai-memory/blob/main/factory.rs) supports any provider implementing the `LlmProvider` trait, including Anthropic, Google Gemini, OpenAI-OAuth, and self-hosted OpenAI-compatible endpoints. Configure them together using `AI_MEMORY_LLM_PROVIDER_CHAIN` with comma-separated provider identifiers.

### How does the fallback chain report errors when all providers fail?

When every candidate in the chain exhausts, the system returns `LlmError::AllCandidatesFailed` with metadata including the total attempt count and a redacted error summary. This preserves diagnostic information while protecting sensitive API keys and tokens from log exposure.