How LLM Providers Are Managed in ai-memory: A Complete Technical Guide

LLM providers in ai-memory are abstracted behind the LlmProvider trait, with eight concrete implementations (OpenAI, Anthropic, Google, Gemini, Copilot, Local, OpenAI OAuth, and OIDC) instantiated through a factory pattern that reads from Figment configuration.

The ai-memory crate orchestrates multiple Large Language Model backends through a unified, async-compatible interface. At the core of this system lies the LlmProvider trait defined in crates/ai-memory-llm/src/provider.rs, which decouples the rest of the codebase from vendor-specific HTTP implementations and authentication mechanisms.

The LlmProvider Trait: Core Abstraction

The LlmProvider trait defines a minimal, provider-agnostic API that all LLM backends must implement. This design enables type-erased, thread-safe provider storage via Arc<dyn LlmProvider> across the MCP server and hook router.

The trait specifies six essential methods:

• name() – Returns a human-readable identifier (e.g., "openai", "anthropic")
• model() – Provides the model name used in HTTP requests
• complete() – Executes plain-text chat completion
• complete_with_operation_id() – Completion tied to a logical operation ID (default fallback to complete)
• complete_structured_raw() – JSON-schema-constrained completion returning serde_json::Value
• complete_structured_raw_with_operation_id() – Operation-aware structured completion (default forwards to raw method)

The trait carries Send + Sync bounds, allowing safe concurrent access from any Tokio task without cloning connection state.

Supported LLM Provider Implementations

ai-memory ships eight concrete implementations, each isolated to its own source file under crates/ai-memory-llm/src/:

Provider	Source File	Key Implementation Details
OpenAI	openai.rs	reqwest client with rustls-tls-native-roots; operation IDs mapped to OpenAI's assistant_id field
Anthropic	anthropic.rs	Messages API endpoint with streaming support and completion_id metadata handling
Google Vertex AI	google.rs	Service-account authentication; model name mapping to Vertex AI identifiers
Gemini	gemini.rs	Native Gemini request format with specialized schema handling
GitHub Copilot	copilot.rs	Copilot completions endpoint supporting both plain-text and structured outputs
Local (in-process)	local.rs	Pure-Rust embeddings via candle crate for offline/test environments
OpenAI OAuth	openai_oauth.rs	Token refresh logic layered over base OpenAI provider
OIDC	oidc.rs	Generic OpenID Connect token handling for OIDC-capable providers

Each implementation handles its own authentication, request serialization, error mapping, and response parsing while exposing the same uniform interface.

Provider Factory Pattern and Configuration

Provider instantiation is centralized through LlmProviderFactory in crates/ai-memory-llm/src/factory.rs. The factory consumes Figment configuration to determine which concrete provider to build, injects appropriate credentials, and returns a boxed dyn LlmProvider.

Standard construction pattern:

let cfg = figment::Figment::new()
    .merge(figment::providers::Toml::file("ai-memory.toml"));

let provider = ai_memory_llm::factory::make_provider(&cfg)?; 
// Returns Box<dyn LlmProvider>

Configuration values—API keys, model names, endpoint URLs, OAuth scopes—are sourced from ai-memory.toml and processed through the core Config struct in crates/ai-memory-core/src/config.rs. This centralization ensures credential isolation and environment-specific overrides without code changes.

Typed Helper Functions for Structured Output

Beyond raw trait methods, ai-memory provides high-level helpers in crates/ai-memory-llm/src/provider.rs (lines 70-106) that automate JSON schema generation and deserialization:

use ai_memory_llm::{provider::LlmProvider, complete_structured};
use ai_memory_llm::types::{ChatRequest, ChatResponse};

let request = ChatRequest::new("Explain quantum tunneling");
let response: MyResponse = complete_structured(&*provider, request).await?;

These helpers leverage schemars for compile-time schema derivation, eliminating manual JSON Schema construction while maintaining full type safety. The complete_structured and complete_structured_with_operation_id functions wrap the raw trait methods, bridging the gap between Rust's type system and LLM JSON outputs.

Thread Safety and Concurrency Model

The Send + Sync trait bounds enable shared provider ownership across async boundaries. A single Arc<dyn LlmProvider> can be:

• Stored in the MCP server state
• Passed to hook routers
• Cloned into spawned Tokio tasks

This design avoids connection-per-request overhead while maintaining safety guarantees. Providers internally manage their own connection pooling and rate limiting where appropriate.

Summary

• Trait abstraction: LlmProvider isolates vendor-specific HTTP logic behind six core methods
• Eight implementations: OpenAI, Anthropic, Google, Gemini, Copilot, Local, OpenAI OAuth, and OIDC each live in dedicated source files
• Factory instantiation: LlmProviderFactory reads Figment configuration to build the correct provider at runtime
• Type-safe helpers: complete_structured functions automate JSON schema handling via schemars
• Thread-safe design: Send + Sync bounds permit Arc<dyn LlmProvider> sharing across Tokio tasks

Frequently Asked Questions

How do I add a new LLM provider to ai-memory?

Implement the LlmProvider trait for your provider struct in a new file under crates/ai-memory-llm/src/, then register it in the factory at crates/ai-memory-llm/src/factory.rs. Your implementation must satisfy Send + Sync and handle its own HTTP client configuration and authentication.

Can I use multiple LLM providers simultaneously in the same ai-memory instance?

Yes. Because providers are trait objects, you can instantiate multiple Box<dyn LlmProvider> or Arc<dyn LlmProvider> values from the factory and route requests to different providers based on operation type, cost optimization, or fallback logic.

Where are API credentials stored in ai-memory?

Credentials reside in the Figment-managed configuration file (ai-memory.toml). The Config struct in crates/ai-memory-core/src/config.rs parses these values and passes them to the factory, keeping secrets out of source code and enabling environment-specific configuration files.

What is the purpose of operation IDs in LLM completions?

Operation IDs enable request tracing and idempotency across distributed systems. The complete_with_operation_id and complete_structured_raw_with_operation_id methods allow callers to associate a logical operation identifier with a completion request, which providers like OpenAI map to native fields (assistant_id) for tracking and potential retry semantics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}