How to Build a Multi-Provider Fallback Mechanism for Agents in aisuite

aisuite enables automatic failover between LLM providers by catching LLMError exceptions from provider instances created via ProviderFactory, allowing your agents to retry requests with backup providers in sequence until one succeeds.

aisuite is a Python library that unifies access to multiple large language model providers through a single interface. When building production-grade AI agents, implementing a multi-provider fallback mechanism ensures high availability by automatically switching to backup providers when the primary experiences outages, rate limits, or errors. This guide demonstrates how to construct robust fallback logic using aisuite's provider-agnostic architecture as implemented in the andrewyng/aisuite repository.

Understanding the Provider Architecture

aisuite uses a standardized naming convention to route requests. The model identifier follows the format <provider>:<model-name>, such as "openai:gpt-4o" or "anthropic:claude-3-5-sonnet-20240620".

Dynamic Provider Loading

At runtime, aisuite.provider.ProviderFactory dynamically loads the appropriate provider class based on the identifier prefix. According to the source code in aisuite/provider.py, the factory locates provider modules following the naming convention <provider>_provider.py and instantiates the corresponding <Provider>Provider class (e.g., OpenaiProvider, AnthropicProvider).

Each concrete provider implements the uniform chat_completions_create method, ensuring consistent interaction regardless of the underlying SDK. When failures occur, providers raise aisuite.provider.LLMError, which serves as the canonical exception for fallback logic to catch.

Method 1: Simple Fallback Wrapper for Direct Calls

For agents making direct client calls, wrap the provider invocation in a sequencer that iterates through a priority list of models. This approach uses ProviderFactory.create_provider(provider_key, config) to instantiate each provider dynamically.

import aisuite as ai
from aisuite.provider import ProviderFactory, LLMError

def chat_with_fallback(model_list, messages, **kwargs):
    """
    Try the models in ``model_list`` in order.
    Returns the first successful response or re-raises the last LLMError.
    """
    last_error = None
    for model in model_list:
        provider_key = model.split(":")[0]                # e.g. "openai"

        provider = ProviderFactory.create_provider(
            provider_key, config={}                       # add any required config here

        )
        try:
            return provider.chat_completions_create(
                model=model, messages=messages, **kwargs
            )
        except LLMError as exc:                           # provider-specific failure

            last_error = exc
    # If we reach here, all providers failed

    raise last_error

Usage example:

client = ai.Client()

models = [
    "openai:gpt-4o",          # primary

    "anthropic:claude-3-5-sonnet-20240620",  # secondary

    "groq:mixtral-8x7b-32768",               # tertiary

]

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": "Summarize the latest news in tech."},
]

response = chat_with_fallback(models, messages, temperature=0.7)
print(response.choices[0].message.content)

This helper abstracts provider selection, allowing any part of your agent code to call chat_with_fallback instead of the standard client.chat.completions.create.

Method 2: Integrated Fallback via Agent Runner Subclassing

For deeper integration with aisuite's Agents API, subclass aisuite.agents.runner.Runner to inject fallback logic directly into the execution flow. This method overrides the internal _call_provider method used by the Runner to make chat requests.

from aisuite.agents.runner import Runner
from aisuite.provider import ProviderFactory, LLMError

class FallbackRunner(Runner):
    def __init__(self, agent, fallback_models, **kwargs):
        super().__init__(agent, **kwargs)
        self.fallback_models = fallback_models   # ordered list of model strings

    def _call_provider(self, model, messages, **kwargs):
        # Override the internal method used by Runner to make a chat request

        for m in self.fallback_models:
            provider_key = m.split(":")[0]
            provider = ProviderFactory.create_provider(provider_key, config={})
            try:
                return provider.chat_completions_create(
                    model=m, messages=messages, **kwargs
                )
            except LLMError:
                continue
        raise RuntimeError("All fallback providers failed.")

Running an agent with fallback:

import aisuite as ai
from aisuite import Agent

agent = Agent(
    name="news-summarizer",
    model="openai:gpt-4o",        # primary model (fallback logic will ignore this)

    instructions="Summarize tech headlines.",
    tools=[],
)

fallback_models = [
    "openai:gpt-4o",
    "anthropic:claude-3-5-sonnet-20240620",
    "groq:mixtral-8x7b-32768",
]

result = FallbackRunner(agent, fallback_models).run(
    "What are the biggest AI announcements this week?"
)
print(result.final_output)

The FallbackRunner swaps providers transparently, keeping the agent definition unchanged while ensuring every turn uses the fallback-aware client.

Method 3: Async Fallback for Concurrent Agents

For asynchronous agent implementations, implement achat_with_fallback using the async variant achat_completions_create. This follows the same pattern as the synchronous version but awaits coroutines.

import aisuite as ai
from aisuite.provider import ProviderFactory, LLMError

async def achat_with_fallback(models, messages, **kwargs):
    last_error = None
    for model in models:
        provider_key = model.split(":")[0]
        provider = ProviderFactory.create_provider(provider_key, config={})
        try:
            return await provider.achat_completions_create(
                model=model, messages=messages, **kwargs
            )
        except LLMError as exc:
            last_error = exc
    raise last_error

Use this function in async agent loops to maintain non-blocking execution while preserving multi-provider resilience.

Key Source Files Reference

Understanding these core files helps when debugging or extending the fallback mechanism:

• aisuite/provider.py: Contains the abstract Provider class, ProviderFactory for dynamic loading, and the LLMError exception definition.
• aisuite/providers/openai_provider.py: Example concrete provider implementing chat_completions_create and achat_completions_create.
• aisuite/providers/anthropic_provider.py: Demonstrates how multiple providers share the uniform interface.
• aisuite/client.py: High-level client that parses the <provider>:<model> identifier and forwards calls to the appropriate provider.
• aisuite/agents/runner.py: Execution engine for agents; designed to be subclassed to inject custom logic like provider fallback.

Summary

• aisuite's provider-agnostic design uses the <provider>:<model> identifier format and ProviderFactory to enable seamless switching between LLM backends.
• Catch LLMError from aisuite/provider.py to detect provider-specific failures and trigger fallback logic.
• Implement fallback by iterating through a priority list of models, splitting each identifier to extract the provider key, and calling ProviderFactory.create_provider to instantiate the correct client.
• Integrate at the Runner level by subclassing aisuite.agents.runner.Runner and overriding _call_provider to make fallback invisible to agent definitions.
• Support async agents by using achat_completions_create with identical error handling patterns.

Frequently Asked Questions

What exception should I catch when implementing a fallback mechanism in aisuite?

Catch aisuite.provider.LLMError. This is the unified exception raised by all concrete provider implementations (such as those in aisuite/providers/openai_provider.py and aisuite/providers/anthropic_provider.py) when API calls fail, time out, or return errors. Catching this specific exception allows your fallback logic to distinguish between provider failures and other application errors.

Can I combine OpenAI and Anthropic models in the same fallback chain?

Yes. Because ProviderFactory.create_provider returns instances that share the same interface, you can chain heterogeneous providers like "openai:gpt-4o", "anthropic:claude-3-5-sonnet-20240620", and "groq:mixtral-8x7b-32768" in a single fallback list. Each provider handles its own authentication and SDK specifics internally while exposing the uniform chat_completions_create method to your agent.

How do I integrate fallback logic into aisuite's Agent Runner?

Subclass aisuite.agents.runner.Runner and override the _call_provider method. Inside your override, implement a loop that attempts the request with each provider in your fallback list, catching LLMError and continuing to the next provider on failure. This injects fallback behavior at the execution engine level, making it transparent to the agent's definition and instructions.

Does aisuite support async fallback for non-blocking agent execution?

Yes. aisuite providers implement achat_completions_create alongside the synchronous chat_completions_create. You can build an async fallback mechanism by awaiting these methods in sequence within an async function, catching LLMError exceptions exactly as you would in synchronous code. This allows multi-provider resilience in concurrent or high-throughput async agent architectures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}