How OmniRoute's 3-Layer Resilience System Protects Against Provider Failures

OmniRoute uses a hierarchical circuit breaker, connection cooldown, and model lockout system to isolate failures at the provider, credential, and model levels, ensuring automatic fallback and high availability.

OmniRoute is an open-source LLM routing proxy that maintains request throughput even when upstream providers fail. Its 3-layer resilience system operates at progressively finer scopes—provider, connection, and model—to contain failures without disrupting healthy traffic paths.

---

Provider Circuit Breaker: Coarse-Grained Failure Isolation

The provider circuit breaker guards against complete provider outages by halting all traffic to a provider that repeatedly returns upstream errors.

Located in src/shared/utils/circuitBreaker.ts, the breaker implements a standard CLOSED → OPEN → HALF_OPEN state machine:

• When failure threshold is hit (e.g., multiple 5xx responses), the breaker flips to OPEN and blocks all requests for a configurable cooldown period
• The next status read after timeout transitions to HALF_OPEN
• A successful probe request returns the breaker to CLOSED; another failure reopens it

This lazy recovery design requires no background timer—state transitions trigger only on demand when canExecute() or getProviderStatus() is called.

// Example: Checking provider status before sending a request
import { getProviderStatus, canExecute } from '@/shared/utils/circuitBreaker';

if (canExecute('openai')) {
  // Provider is CLOSED or HALF_OPEN – safe to use
  await sendChatRequest(...);
} else {
  // Provider is OPEN – skip or use fallback
  await fallbackToAnotherProvider();
}

The circuit breaker prevents cascade failures by giving providers time to recover before traffic resumes.

---

Connection Cooldown: Credential-Level Recovery

The connection cooldown layer protects individual credentials (API keys, accounts) within a provider, allowing healthy keys to continue serving traffic while problematic ones rest.

Implementation spans two files:

• src/sse/services/auth.ts::markAccountUnavailable() — marks failed connections with rateLimitedUntil timestamp and testStatus = "unavailable"
• open-sse/services/accountFallback.ts::checkFallbackError() — filters unavailable connections during credential selection

When a request fails with recoverable errors (429, 500), the connection receives an exponential back-off penalty:

// Example: Marking a connection as unavailable after a 429 response
import { markAccountUnavailable } from '@/sse/services/auth';

async function handleResponse(res: Response, connId: string) {
  if (res.status === 429) {
    // Apply exponential back-off: baseCooldownMs * 2 ** failureIndex
    await markAccountUnavailable(connId, {
      errorCode: 429,
      backoffLevel: 1, // incremented internally
    });
  }
}

The cooldown formula baseCooldownMs * 2 ** failureIndex ensures aggressive back-off for repeatedly failing keys. Like the circuit breaker, recovery is lazy—once rateLimitedUntil passes, the connection becomes eligible again without active reconciliation.

---

Model Lockout: Fine-Grained Quota Protection

The model lockout layer isolates failures to specific models, preventing a single model's quota exhaustion or permission error from disabling an entire connection.

Implemented in open-sse/services/accountFallback.ts, this mechanism handles model-specific errors:

• 429 quota exceeded on a particular model
• 404 model not found or permission denied

Only the affected model is flagged; the same connection continues serving other models.

// Example: Locking out a single model after a quota error
import { lockModel } from '@/open-sse/services/accountFallback';

if (error.code === 429 && error.model === 'gpt-4') {
  await lockModel('openai', connectionId, 'gpt-4');
}

This granular containment preserves capacity across the model portfolio and avoids over-reaction to transient quota limits on popular models.

---

How the Three Layers Work Together

OmniRoute's resilience system forms a hierarchical failure containment strategy:

Priority	Layer	Failure Scope	Response Speed
1	Circuit Breaker	Entire provider	Fastest—immediate cut-off
2	Connection Cooldown	Single credential	Medium—back-off and retry
3	Model Lockout	Specific model	Granular—preserve connection utility

When a provider fails completely, the circuit breaker triggers first, routing traffic to alternate providers via combo routing. If only one key within a provider fails, connection cooldown preserves other keys. If a single model hits quota, model lockout keeps the connection alive for remaining models.

All three mechanisms respect upstream back-off signals while maintaining automatic fallback through OmniRoute's routing layer.

---

Summary

• Provider circuit breaker (src/shared/utils/circuitBreaker.ts) stops all traffic to unhealthy providers using lazy state transitions
• Connection cooldown (src/sse/services/auth.ts, open-sse/services/accountFallback.ts) applies exponential back-off to individual credentials while preserving sibling keys
• Model lockout (open-sse/services/accountFallback.ts) isolates per-model failures without disabling entire connections
• Combined, these layers enable automatic fallback, high availability, and respectful back-off toward upstream services

---

Frequently Asked Questions

How quickly does the circuit breaker react to provider failures?

The circuit breaker updates state immediately upon threshold violation. Since it uses lazy recovery, no polling overhead exists—status checks occur only when canExecute() evaluates a routing decision. Typical detection-to-isolation latency is one failed request plus the threshold count.

What happens if all providers in a combo route are OPEN?

When all configured providers are circuit-broken, OmniRoute returns a 503 Service Unavailable with diagnostic metadata indicating all breaker states. Administrators can configure dead-letter routing or queue-and-retry policies in docs/architecture/RESILIENCE_GUIDE.md.

Does connection cooldown persist across server restarts?

By default, cooldown state is in-memory only and resets on restart. For distributed deployments, Redis-backed state can be enabled in src/sse/services/auth.ts by configuring a shared cache provider.

Can model lockout be configured for specific error codes?

Yes. The lockModel() function in open-sse/services/accountFallback.ts accepts configurable error predicates. Modify checkFallbackError() to include additional codes (e.g., 402 payment required) or exclude certain transient errors from triggering lockout.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}