OmniRoute Resilience System: Understanding the Three Layers of Failure Isolation

OmniRoute isolates transient failures using three distinct runtime mechanisms—Provider Circuit Breaker, Connection Cooldown, and Model Lockout—each operating at different granularity levels to prevent cascading errors across the LLM routing pipeline.

The diegosouzapw/OmniRoute repository implements a multi-tiered OmniRoute resilience system that protects AI request pipelines from provider outages, rate limits, and transient errors. By compartmentalizing failures at the provider, credential, and model levels, the system ensures high availability even when upstream services degrade. This architecture leverages lazy-recovery strategies and automatic state transitions to minimize manual intervention while preventing noisy failures from impacting end users.

1. Provider Circuit Breaker: Whole-Provider Protection

The Provider Circuit Breaker operates at the broadest scope, monitoring entire LLM providers (e.g., openai, anthropic, glm) for fatal upstream errors such as 5xx responses or timeouts. When a provider exhibits repeated failures, the breaker transitions to the OPEN state, halting all traffic to that provider until health checks confirm recovery.

Implementation Details

The core circuit breaker logic resides in src/shared/utils/circuitBreaker.ts, which maintains state machines for each configured provider. The SSE chat handlers in src/sse/handlers/chatHelpers.ts consult this utility before routing requests, checking whether a provider’s breaker is closed, open, or half-open. This design prevents requests from being wasted on unhealthy upstream services during outages.

Monitoring via Health Endpoint

Operators can inspect breaker status through the dedicated health endpoint defined in src/app/api/monitoring/health/route.ts. This route returns JSON objects describing each provider’s current state, including retry timestamps.

// Query the health API to check provider circuit status
// Returns: { "openai": { "state": "OPEN", "retryAfterMs": 120000 } }
await fetch('http://localhost:20128/api/monitoring/health')
  .then(r => r.json())
  .then(status => console.log(status.openai));

When the breaker detects a network outage causing 502/503 errors, it automatically moves to OPEN and routes traffic to alternative providers until a probe succeeds.

2. Connection Cooldown: Per-Credential Suspension

The Connection Cooldown layer provides finer-grained control by isolating individual credentials or API keys within a provider while allowing other accounts to remain active. This mechanism handles transient issues like rate limiting (429 responses) without blacklisting the entire provider.

Account Unavailability Logic

Credential state management occurs in src/sse/services/auth.ts, specifically through the markAccountUnavailable function. When a specific key encounters errors, this function sets availability flags and timestamps (such as rateLimitedUntil) that the fallback system evaluates before selecting a routing target. The checkFallbackError function in open-sse/services/accountFallback.ts performs these evaluations, calculating backoff periods based on error types and retry counts.

Programmatic Cooldown Management

Administrators can manually trigger cooldowns for testing or emergency maintenance using the auth service utilities:

import { markAccountUnavailable } from '@/sse/services/auth';

// Trigger cooldown after detecting a 429 on a specific key
await markAccountUnavailable({
  provider: 'openai',
  credentialId: 'key-abc123',
  errorCode: 429,
  backoffLevel: 1,
});

This approach ensures that when one API key hits a quota limit, traffic immediately fails over to other valid keys for the same provider without service interruption.

3. Model Lockout: Granular Model-Level Isolation

The Model Lockout layer offers the most precise isolation, targeting specific model identifiers (e.g., gpt-4o-preview) that return errors while preserving access to other models on the same credential. This addresses per-model quotas and temporary model unavailability scenarios.

Fallback Error Handling

Model-specific lockouts are managed within the same fallback logic in open-sse/services/accountFallback.ts. When a request to a particular model fails due to quota exhaustion or missing model errors, the system sets a model-specific lockout flag rather than suspending the entire connection. This flag is checked during subsequent routing decisions, ensuring only the problematic model is bypassed.

Clearing Model-Specific Lockouts

The system stores lockout states in credential objects alongside recovery timestamps. You can programmatically detect and clear these states when conditions improve:

import { getProviderCredentials } from '@/sse/services/auth';
import { clearModelLockout } from '@/open-sse/services/accountFallback';

const creds = await getProviderCredentials('openai', 'key-abc123');
if (creds.modelLockout?.has('gpt-4o-preview')) {
  // Force immediate retry after manual quota increase
  await clearModelLockout('openai', 'key-abc123', 'gpt-4o-preview');
}

This granular approach prevents a single model’s quota issues from forcing traffic onto entirely different providers when other models on the same key remain viable.

Shared Recovery Strategy

All three layers of the OmniRoute resilience system implement a lazy-recovery pattern rather than active polling. Each layer stores expiration timestamps—such as rateLimitedUntil for cooldowns and resetAt for circuit breakers—and automatically promotes states to HALF_OPEN or clears flags upon the first request after expiration. This design eliminates stale blocks and reduces overhead while ensuring the system remains responsive to genuine recovery events.

Summary

• Provider Circuit Breaker (src/shared/utils/circuitBreaker.ts) protects entire LLM providers from cascading failures by moving unhealthy providers to an OPEN state.
• Connection Cooldown (src/sse/services/auth.ts) suspends individual API keys or accounts experiencing rate limits while preserving provider access through alternate credentials.
• Model Lockout (open-sse/services/accountFallback.ts) isolates specific models suffering quota exhaustion without affecting other models on the same connection.
• All layers use lazy-recovery with timestamp-based expiration to automatically restore service without manual intervention.
• The health endpoint (src/app/api/monitoring/health/route.ts) exposes real-time breaker status for operational monitoring.

Frequently Asked Questions

What triggers the Provider Circuit Breaker to open?

The breaker opens when a provider repeatedly returns fatal upstream errors such as 5xx status codes or timeouts. According to the implementation in src/shared/utils/circuitBreaker.ts, the system tracks failure thresholds and transitions to OPEN when consecutive errors exceed configured limits, typically during network outages or provider degradation.

How long does a Connection Cooldown last?

Cooldown duration depends on the error type and backoff level specified when calling markAccountUnavailable in src/sse/services/auth.ts. Rate-limited connections (429 errors) use exponential backoff calculations stored in timestamps like rateLimitedUntil, while the actual period varies based on the backoffLevel parameter passed during the initial marking.

Can I manually clear a Model Lockout before the automatic recovery?

Yes. While the system automatically clears lockouts based on expiration timestamps, you can force immediate recovery using clearModelLockout from open-sse/services/accountFallback.ts. This is useful after manually increasing quotas or resolving upstream issues, allowing you to restore service without waiting for the lazy-recovery cycle.

How do I monitor the current state of all three resilience layers?

Query the health endpoint at /api/monitoring/health (implemented in src/app/api/monitoring/health/route.ts) for provider-level circuit breaker states. For connection and model-level details, inspect credential objects via getProviderCredentials in src/sse/services/auth.ts, which exposes availability flags and model-specific lockout sets for programmatic monitoring and alerting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}