How to Configure Multi-Provider Fallback Chains with OmniRoute Routing Strategies

OmniRoute enables resilient LLM request handling by allowing you to define ordered provider sequences called combos that automatically failover to secondary providers when primaries fail, controlled by declarative routing strategies.

OmniRoute is an open-source LLM gateway that abstracts multiple providers behind a unified interface. By configuring multi-provider fallback chains, you ensure high availability for AI-powered applications even when individual providers experience rate limits or outages.

Understanding Combo-Based Routing Architecture

OmniRoute implements failover logic through three core primitives: combos, routing strategies, and resolvers. Understanding these components is essential before implementing fallback chains.

The Combo Abstraction

A combo is a named collection of provider-model pairs stored in the database layer at src/lib/db/combo.ts. Each combo entry specifies a target provider, model identifier, and optional credentials. When a request arrives, OmniRoute expands this collection into executable targets using resolveComboTargets() before attempting execution.

Combos are validated against Zod schemas defined in src/shared/validation/schemas/combo.ts, which enforce that strategy names match the canonical list exported from src/shared/constants/routingStrategies.ts.

Routing Strategies Overview

The ROUTING_STRATEGY_VALUES constant in src/shared/constants/routingStrategies.ts defines how OmniRoute orders and selects targets within a combo:

• priority: Attempts targets in the exact order defined in the configuration, creating a strict fallback chain.
• fill-first: Exhausts the first target's quota before moving to subsequent entries.
• weighted: Selects targets probabilistically based on assigned weight values, retrying with remaining candidates on failure.
• least-used: Routes to the provider with the lowest recent invocation count.
• cost-optimized: Automatically selects the cheapest provider that satisfies the token budget constraints.

Defining Fallback Chains in Practice

Configuring a robust fallback chain requires creating a combo definition and attaching the appropriate strategy to handle failure scenarios.

Creating a Combo Configuration

Define your provider sequence as a JSON object where each target includes the provider identifier and model name. The following example establishes a three-tier fallback chain:

{
  "name": "production-fallback",
  "strategy": "priority",
  "targets": [
    { "provider": "openai", "model": "gpt-4o-mini" },
    { "provider": "anthropic", "model": "claude-3-sonnet-20240229" },
    { "provider": "groq", "model": "llama3-70b-8192" }
  ]
}

When using the weighted strategy, include a weight property to influence selection probability:

{
  "name": "balanced-fallback",
  "strategy": "weighted",
  "targets": [
    { "provider": "openai", "model": "gpt-4o-mini", "weight": 5 },
    { "provider": "anthropic", "model": "claude-3-haiku-20240307", "weight": 3 },
    { "provider": "groq", "model": "mixtral-8x7b-32768", "weight": 2 }
  ]
}

Selecting the Appropriate Strategy

Choose priority when you need deterministic failover behavior, ensuring requests only reach secondary providers when primaries return non-retryable errors. Use weighted for load distribution across healthy providers while maintaining implicit fallback capabilities. The cost-optimized strategy suits batch processing workloads where latency is less critical than expense management.

Implementation Details from the Source Code

OmniRoute's fallback mechanism operates within the SSE (Server-Sent Events) service layer, specifically in open-sse/services/combo.ts.

Resolution and Execution Flow

When handling a chat completion request, the system executes the following sequence:

1. The route handler extracts the combo name from the request body or user profile defaults.
2. resolveComboTargets() queries src/lib/db/combo.ts to retrieve the combo definition and expand it into a ResolvedComboTarget[] array.
3. handleComboChat() receives the resolved array and the selected strategy, then iterates over targets according to the strategy's ordering algorithm.
4. The Translator layer converts provider-specific response formats back to the client-facing schema.

Fallback Trigger Mechanism

The resolver advances to the next target when it encounters specific error conditions. According to the implementation in open-sse/services/combo.ts, fallback triggers include HTTP 429 (rate limit), HTTP 500 (server error), and provider-specific "model unavailable" responses. This ensures that transient provider failures automatically activate your configured backup providers without client-side intervention.

Configuration Examples

You can persist multi-provider fallback chains using either the command-line interface or the REST API endpoints located in src/app/api/settings/combo/.

CLI-Based Setup

Create a JSON file containing your combo definition, then register it using the OmniRoute CLI:

cat > production-fallback.json <<'EOF'
{
  "name": "production-fallback",
  "strategy": "priority",
  "targets": [
    { "provider": "openai", "model": "gpt-4o-mini" },
    { "provider": "anthropic", "model": "claude-3-sonnet-20240229" },
    { "provider": "groq", "model": "llama3-70b-8192" }
  ]
}
EOF

omniroute combo set --file production-fallback.json

Once registered, any request specifying "combo": "production-fallback" will attempt OpenAI first, fall back to Anthropic on failure, and finally try Groq if both previous providers are unavailable.

REST API Configuration

For programmatic management, use the Settings API endpoint. The following cURL command creates a weighted fallback chain:

curl -X POST https://your-omniroute.domain/api/settings/combo \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMNIRUTE_API_KEY" \
  -d '{
    "name": "api-weighted-fallback",
    "strategy": "weighted",
    "targets": [
      { "provider": "openai", "model": "gpt-4o-mini", "weight": 7 },
      { "provider": "anthropic", "model": "claude-3-haiku-20240307", "weight": 3 }
    ]
  }'

The API validates the payload against the Zod schemas in src/shared/validation/schemas/combo.ts, ensuring the strategy value exists in ROUTING_STRATEGY_VALUES before persistence.

Monitoring and Optimization

OmniRoute exposes combo performance metrics through src/lib/db/comboMetrics.ts, allowing you to identify providers that frequently trigger fallbacks. The list_combo_metrics MCP tool defined in open-sse/mcp-server/schemas/tools.ts queries success and failure counts per combo entry.

For cost-sensitive workloads, enable the cost-optimized strategy and update provider pricing data via src/lib/pricingSync.ts. This configuration automatically routes requests to the cheapest available provider while maintaining your fallback chain for reliability.

Summary

• Combos are named provider collections stored in src/lib/db/combo.ts that define your fallback candidate pool.
• Routing strategies (priority, weighted, cost-optimized, etc.) in src/shared/constants/routingStrategies.ts determine failover order and selection logic.
• The resolver in open-sse/services/combo.ts executes handleComboChat() to iterate through targets when errors occur.
• Configure chains via CLI (omniroute combo set) or REST API (POST /api/settings/combo) using validated JSON schemas.
• Monitor fallback frequency using combo metrics to optimize provider selection and cost efficiency.

Frequently Asked Questions

What error codes trigger a fallback to the next provider in the chain?

According to open-sse/services/combo.ts, OmniRoute advances to the next target when it encounters HTTP 429 (rate limit), HTTP 500 (server errors), or provider-specific "model unavailable" responses. Standard retry logic applies to transient network errors, but non-retryable provider errors immediately activate the fallback mechanism.

How does the weighted strategy handle provider failures?

When using the weighted strategy, the resolver first selects a target based on the probability distribution defined by weight values. If that provider fails, the system removes it from the current iteration pool and reselects from remaining targets using the same weight ratios. This ensures failed providers are skipped while maintaining load distribution across healthy alternatives.

Can I use different routing strategies for different types of requests?

Yes. OmniRoute stores combos as independent configurations in the database, and each request specifies its desired combo via the request body or user profile defaults. You can create separate combos with different strategies—such as a "priority" combo for real-time chat and a "cost-optimized" combo for batch processing—and route requests accordingly through the src/app/api/settings/combo/* endpoints.

Where is the list of valid routing strategy values defined?

The canonical list of strategy identifiers is exported as ROUTING_STRATEGY_VALUES from src/shared/constants/routingStrategies.ts. This array is referenced by the Zod validation schemas in src/shared/validation/schemas/combo.ts to ensure only supported strategies can be persisted through the API or CLI interfaces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}