How OmniRoute's 4-Tier Fallback System Prioritizes Providers for LLM Routing

OmniRoute uses a deterministic four-step fallback ladder that isolates failures at the provider, connection, model, and policy levels before selecting the next viable provider from a priority-sorted chain.

OmniRoute's provider prioritization is implemented as a cascading filter system in the diegosouzapw/OmniRoute repository. Each tier guards a different failure domain, ensuring that transient or localized issues don't trigger unnecessary provider switches while still maintaining high availability for LLM requests.

Tier 1: Provider Circuit Breaker

The provider circuit breaker operates at the highest level, guarding entire provider backends (e.g., OpenAI, Anthropic) from being routed to when they are experiencing widespread issues.

In src/shared/utils/circuitBreaker.ts, the breaker tracks upstream errors (408, 500‑504) and flips to OPEN state after a configurable threshold. When open, the provider is excluded from all routing decisions immediately.

This prevents cascading failures and gives troubled providers time to recover before traffic resumes.

Tier 2: Connection Cooldown

Individual API keys or accounts within a provider can hit rate limits independently. The connection cooldown system in src/sse/services/auth.ts marks connections with a rateLimitedUntil timestamp when they return transient errors (429, 503).

Connections whose cooldown timestamp is still in the future are skipped during provider selection, allowing other healthy connections from the same provider to continue serving requests.

Tier 3: Model Lockout

Some failures are model-specific rather than connection-wide. The model lockout mechanism in open-sse/services/accountFallback.ts isolates individual models on specific connections when quota or permission errors occur.

This granularity ensures that hitting a limit on gpt-4o doesn't disable the connection entirely—other models on the same key remain usable.

Tier 4: Declarative Fallback Chain

After the first three tiers filter out unavailable providers, OmniRoute consults the fallback chain for the target model. This priority-sorted list is defined in src/domain/fallbackPolicy.ts and maps model names to ordered provider arrays.

Lower priority values indicate higher precedence. The first enabled provider not already excluded by upstream tiers is selected:

import {
  resolveFallbackChain,
  registerFallback,
  getNextFallback,
} from '@/domain/fallbackPolicy';

// Register priority order for gpt-4o
registerFallback('gpt-4o', [
  { provider: 'openai', priority: 0 },
  { provider: 'anthropic', priority: 1 },
  { provider: 'cohere', priority: 2, enabled: false }, // bypassed
]);

// After circuit breaker, cooldown, and lockout checks
const excluded = ['openai'];  // already attempted
const next = getNextFallback('gpt-4o', excluded);

if (next) {
  await dispatchToProvider(next, request);
} else {
  throw new Error('All fallback providers exhausted');
}

The Complete Prioritization Flow

OmniRoute applies these tiers in strict sequence for every request:

1. Circuit-breaker check — remove providers with OPEN breakers
2. Connection-cooldown check — skip connections in rate-limit cooldown
3. Model-lockout check — exclude model+connection pairs that are locked
4. Fallback-chain lookup — select the lowest-priority enabled provider from the remaining set

If the chain exhausts without finding a viable provider, the request fails with a "fallback exhausted" error propagated to the client.

Integrating Tier Checks in Request Handlers

The lower-level guards are evaluated before chain resolution:

// From src/shared/utils/circuitBreaker.ts
if (circuitBreaker.isOpen('openai')) {
  // Entire provider excluded
}

// From src/sse/services/auth.ts
if (isConnectionOnCooldown(conn)) {
  // Specific key/account excluded
}

// From open-sse/services/accountFallback.ts
if (isModelLockedOut(conn, 'gpt-4o')) {
  // Specific model on this connection excluded
}

Summary

• Provider circuit breakers guard entire backends against systemic failures
• Connection cooldowns isolate rate-limited keys without disabling providers
• Model lockouts provide fine-grained granularity for quota violations
• Declarative fallback chains determine final provider priority after all filters apply

Each tier in OmniRoute's 4-tier fallback system narrows the candidate pool before the next evaluation, ensuring deterministic, reproducible routing decisions under failure conditions.

Frequently Asked Questions

How does OmniRoute handle multiple rate limits on the same provider?

OmniRoute handles multiple rate limits through connection-level isolation. Each API key or account tracks its own rateLimitedUntil timestamp in src/sse/services/auth.ts. When one connection hits a 429, only that specific connection enters cooldown—other connections from the same provider remain available for routing.

What happens if all providers in a fallback chain fail?

If all providers in a fallback chain are excluded by circuit breakers, cooldowns, lockouts, or prior attempts, OmniRoute throws a "fallback exhausted" error from src/domain/fallbackPolicy.ts. This error propagates to the client, indicating that no viable provider exists for the requested model at that moment.

Can fallback chains be modified at runtime?

Yes. The registerFallback() function in src/domain/fallbackPolicy.ts allows dynamic registration and updates of fallback chains. Each model can have its priority list adjusted without restarting the service, though existing in-flight requests use the chain state at the time of their evaluation.

What's the difference between a circuit breaker and a connection cooldown?

The circuit breaker (src/shared/utils/circuitBreaker.ts) operates at the provider level and responds to server-side errors (500s, timeouts) affecting the entire backend. The connection cooldown (src/sse/services/auth.ts) operates at the key/account level and responds to client-side rate limits (429s) affecting only specific credentials.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}