How ai-memory Manages Multiple LLM Providers with Automatic Fallback Chains

ai-memory unifies diverse LLM backends behind a single FallbackLlmProvider wrapper that implements the LlmProvider trait, automatically cycling through configured providers when transient errors occur while using circuit-breaker logic to skip temporarily unhealthy endpoints.

The ai-memory project by akitaonrails delivers a resilient abstraction layer for large language model integration. Instead of locking users to a single vendor, the system supports configuring multiple LLM providers in a prioritized chain, ensuring your application remains operational even when individual services experience downtime. This architecture centers on the LlmProvider trait defined in crates/ai-memory-llm/src/provider.rs, which standardizes interactions across diverse backends including Anthropic, Google Gemini, and OpenAI-compatible endpoints.

The LlmProvider Trait Abstraction

All LLM implementations in ai-memory conform to the LlmProvider trait defined in crates/ai-memory-llm/src/provider.rs. This interface abstracts provider-specific details, exposing uniform methods like complete and complete_with_operation_id that the rest of the application uses for tasks such as consolidation, linting, and query reranking.

By coding against this trait rather than concrete implementations, ai-memory achieves polymorphic behavior where the consumer cannot distinguish between a single provider and an entire fallback chain. The trait contract requires implementations to return standardized LlmError types, which include the critical is_transient() method used by the fallback logic to distinguish retryable failures from configuration errors.

Configuring the Provider Chain

The CLI reads the llm_provider_chain configuration key at startup to construct an ordered list of providers. Located in crates/ai-memory-llm/src/factory.rs, the build_provider function parses this configuration and instantiates concrete provider implementations—whether Anthropic's Claude, Google's Gemini, or self-hosted OpenAI-compatible endpoints—wrapping them in Candidate structs that maintain health state.

Environment variables control this behavior at runtime:

export AI_MEMORY_LLM_PROVIDER_CHAIN="anthropic,gemini"
export AI_MEMORY_LLM_MODEL="claude-haiku-4-5,gemini-3.5-flash"
export AI_MEMORY_LLM_COOLDOWN_SECS=30

Each entry in the provider chain maps to a specific model configuration, allowing fine-grained control over which model variant serves as primary, secondary, or tertiary options.

FallbackLlmProvider Implementation

The FallbackLlmProvider struct in crates/ai-memory-llm/src/fallback.rs orchestrates the multi-provider logic. It maintains an internal Vec<Candidate> and implements the same LlmProvider trait as individual backends, making it a drop-in replacement throughout the codebase.

Ordered Attempt Logic

When processing a request, the fallback provider iterates sequentially over self.entries. For each request method like complete or complete_with_operation_id, the implementation attempts the operation on the first candidate. Only if that attempt returns a transient error does the loop advance to the next candidate in the chain.

Transient vs Deterministic Error Handling

Error handling distinguishes between recoverable and non-recoverable failures through the LlmError::is_transient() method. Transient errors—such as network timeouts or 5xx server errors—trigger fallback to the next provider. Deterministic errors—such as 400 bad requests, 401 authentication failures, or schema violations—cause immediate short-circuiting of the entire chain.

As implemented in fallback.rs lines 93-95:

if !transient {
    return Err(err);
}

This prevents wasting API calls on requests that will fail identically across all providers.

Circuit Breaker Pattern

To prevent hammering unhealthy providers, ai-memory implements a circuit breaker with a default cooldown of 30 seconds (CIRCUIT_COOLDOWN). When a candidate records a transient error via record_error, the code sets circuit_open_until to 30 seconds in the future. The circuit_open method checks this timestamp and skips candidates whose circuits remain open.

Successful calls immediately clear the circuit through record_success, which resets circuit_open_until to None. This allows fast recovery detection when a provider returns to health.

Health Monitoring

Each candidate maintains detailed health metrics accessible via candidate_health(). Defined in crates/ai-memory-llm/src/health.rs, the CandidateHealth struct exposes:

• last_success_at: Timestamp of the last successful request
• last_error_class: Classification of the most recent failure
• last_error_status: HTTP status code of the last error
• circuit_open_until: Optional timestamp indicating when the circuit closes

Administrators can query these metrics to observe provider reliability and tune the fallback chain configuration.

Practical Usage Examples

Building a fallback chain programmatically follows this pattern:

use ai_memory_llm::fallback::{FallbackLlmProvider, Candidate};
use std::sync::Arc;

let providers = vec![
    Candidate::new(
        "anthropic",
        "claude-haiku-4-5",
        Arc::new(ai_memory_llm::anthropic::AnthropicProvider::new()),
    ),
    Candidate::new(
        "gemini",
        "gemini-3.5-flash",
        Arc::new(ai_memory_llm::gemini::GeminiProvider::new()),
    ),
];

let llm = FallbackLlmProvider::new(providers);

Once constructed, usage is identical to single providers:

let request = ChatRequest::user_prompt("Explain Rust ownership")
    .with_max_tokens(500);
let response = llm.complete(request).await?;
println!("Response: {}", response.text);

When all candidates exhaust, the system returns LlmError::AllCandidatesFailed, aggregating attempt counts and redacted error summaries for debugging without exposing sensitive tokens.

Summary

• ai-memory abstracts LLM interactions through the LlmProvider trait in provider.rs, enabling polymorphic provider handling.
• The FallbackLlmProvider in fallback.rs wraps multiple candidates in a prioritized chain with automatic failover.
• Only transient errors trigger fallback; deterministic errors like authentication failures short-circuit immediately.
• A 30-second circuit breaker prevents retry storms against unhealthy providers, resetting automatically on success.
• Comprehensive health metrics per candidate support operational monitoring and debugging.
• Configuration through environment variables like AI_MEMORY_LLM_PROVIDER_CHAIN enables runtime customization without code changes.

Frequently Asked Questions

How does ai-memory decide when to switch to the next LLM provider?

The FallbackLlmProvider iterates through its internal Vec<Candidate> sequentially. When a provider returns an error, the code checks err.is_transient(). If the error is transient (e.g., network timeout, 5xx status), the loop continues to the next candidate. Non-transient errors abort the chain immediately.

What is the default cooldown period for failed providers?

The default CIRCUIT_COOLDOWN is 30 seconds, configurable via the AI_MEMORY_LLM_COOLDOWN_SECS environment variable. When a provider fails with a transient error, the circuit breaker marks it as unhealthy for this duration before allowing retry attempts.

Can I mix commercial and self-hosted LLM providers in the same chain?

Yes. The factory pattern in factory.rs supports any provider implementing the LlmProvider trait, including Anthropic, Google Gemini, OpenAI-OAuth, and self-hosted OpenAI-compatible endpoints. Configure them together using AI_MEMORY_LLM_PROVIDER_CHAIN with comma-separated provider identifiers.

How does the fallback chain report errors when all providers fail?

When every candidate in the chain exhausts, the system returns LlmError::AllCandidatesFailed with metadata including the total attempt count and a redacted error summary. This preserves diagnostic information while protecting sensitive API keys and tokens from log exposure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}