OmniRoute 3-Layer Resilience System: Circuit Breakers, Connection Cooldown, and Model Lockout

OmniRoute implements a hierarchical 3-layer resilience system using provider-wide circuit breakers, per-failure-kind connection cooldowns, and model-level lockouts to isolate upstream failures and prevent cascading outages.

OmniRoute is an open-source LLM routing framework that maintains high availability through granular fault isolation. The 3-layer resilience system operates at distinct scopes—entire providers, specific error types, and individual model connections—to ensure that transient failures or rate limits do not compromise the entire routing pipeline. According to the diegosouzapw/OmniRoute source code, these protective mechanisms are implemented across circuitBreaker.ts, lockoutPolicy.ts, and the account-fallback service.

Layer 1: Provider-Wide Circuit Breaker

The first line of defense guards all traffic destined for a specific provider (e.g., OpenAI, Gemini) regardless of which model is requested. This layer is implemented in [src/shared/utils/circuitBreaker.ts](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and tracks consecutive failures using a deterministic state machine.

The breaker maintains four states with the following transitions:

• CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED

When the failure count exceeds the configurable failureThreshold (default 5), the breaker transitions to OPEN, immediately short-circuiting further requests to that provider. After the resetTimeout duration elapses, the state shifts to HALF_OPEN, allowing a limited number of probe requests defined by halfOpenRequests. Successful probes restore the CLOSED state, while failed probes reopen the circuit. This prevents "thundering herd" scenarios during upstream outages.

Layer 2: Connection Cooldown (Per-Failure-Kind Back-Off)

The second layer provides fine-grained throttling within the same CircuitBreaker class by applying per-failure-kind cooldowns rather than uniform delays. This prevents aggressive retry loops when a provider returns specific error types like rate limits or quota exhaustion.

The implementation uses a cooldownByKind map and classifyError logic to categorize failures (e.g., rate_limit, quota_exhausted, transient). Each kind can override the generic resetTimeout with a custom duration. The breaker automatically selects the most restrictive timeout via _effectiveCooldown based on the last failure’s classification. For example, a generic 30-second timeout might extend to 120 seconds specifically for HTTP 429 responses, ensuring respectful back-off behavior without penalizing the entire provider for transient network blips.

Layer 3: Model-Level Lockout

The third layer isolates specific model-connection pairs (e.g., gemini-1.5-flash on a specific account) that exhibit repeated failures, without disabling the entire provider. This granularity is handled by [src/domain/lockoutPolicy.ts](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/lockoutPolicy.ts) and consumed by the account-fallback service at [src/services/accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/main/src/services/accountFallback.ts).

The lockout mechanism counts failures per unique identifier within a sliding attemptWindowMs. When failures exceed maxAttempts, a lockout record is persisted to the domain_lockout_state table for the configured lockoutDurationMs. While locked, the model is excluded from combo resolution and routing selection. The lockout clears automatically upon a successful request via recordSuccess or manually through forceUnlock. This ensures that a single misbehaving model cannot degrade the entire provider’s circuit while preserving other healthy endpoints on the same account.

How the Resilience Layers Interact

The three layers operate hierarchically to provide defense in depth:

1. Circuit-Breaker Evaluation: Before any request executes, the system checks if the provider’s circuit is OPEN. If so, the provider is excluded entirely from combo resolution, failing fast to alternatives.

2. Cooldown Application: When the circuit is CLOSED or DEGRADED, the connection cooldown layer may still impose a delay. The next request to that provider is held according to the _effectiveCooldown derived from the last error classification, preventing hammering during rate-limit scenarios.

3. Lockout Consultation: Even when a provider is healthy, the routing logic consults the lockout policy via the account-fallback service. If a specific model-connection pair is locked, it is skipped during target resolution, ensuring requests route only to viable endpoints.

Together, these layers provide provider-wide protection against systemic outages, failure-type throttling for respectful retry behavior, and model-specific isolation for granular fault containment.

Implementation Examples

Configuring a Circuit Breaker with Per-Kind Cooldowns

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Initialize breaker for OpenAI with custom rate-limit handling
const breaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,      // 30 seconds base cooldown
  halfOpenRequests: 1,
  // Aggressive back-off for rate limits specifically
  cooldownByKind: { rate_limit: 120_000 },
});

async function callOpenAI(payload: any) {
  if (!breaker.canExecute()) {
    throw new Error('OpenAI is currently unavailable (circuit open)');
  }

  return breaker.execute(async () => {
    // Actual HTTP fetch implementation
    return fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      body: JSON.stringify(payload),
    });
  });
}

Applying Model-Level Lockouts

import { markAccountUnavailable } from '@/services/accountFallback';

// Trigger lockout after repeated 429 errors for a specific model
await markAccountUnavailable({
  provider: 'gemini',
  connectionId: 'conn-123',
  model: 'gemini-1.5-flash',
  reason: 'rate_limit',
});

// Subsequent combo resolution automatically skips locked models
const combo = await resolveComboTargets(...);

Monitoring Resilience State

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';
import { getAllModelLockouts } from '@/domain/lockoutPolicy';

// Export current health for dashboards or alerting
console.log('Provider breakers:', getAllCircuitBreakerStatuses());
console.log('Model lockouts:', await getAllModelLockouts());

Summary

• Provider-Wide Circuit Breaker: Implements a state machine (CLOSED → DEGRADED → OPEN → HALF_OPEN) in circuitBreaker.ts to cut off all traffic to failing providers after a configurable failure threshold.
• Connection Cooldown: Uses the cooldownByKind map within the same breaker to apply specific back-off durations for different error classifications, preventing aggressive retries during rate limits.
• Model-Level Lockout: Persists failure counts per model-connection pair via lockoutPolicy.ts, temporarily excluding specific endpoints from routing while keeping the provider circuit healthy.
• Hierarchical Defense: The layers cascade from broad provider exclusion to specific error throttling to granular model isolation, ensuring high availability across the OmniRoute pipeline.

Frequently Asked Questions

What causes the circuit breaker to transition from CLOSED to OPEN?

The breaker opens when consecutive failures exceed the failureThreshold parameter (default 5). Each failed request increments an internal counter; when the threshold is crossed, the state becomes OPEN and remains so for the resetTimeout duration before entering HALF_OPEN mode to test recovery.

How does connection cooldown differ from the circuit breaker timeout?

While the circuit breaker’s resetTimeout defines how long the entire provider remains unavailable after an open event, connection cooldown operates within closed circuits to delay specific request types. The cooldownByKind configuration allows longer back-offs for rate limits (e.g., 120 seconds) compared to transient errors (e.g., 5 seconds), whereas the breaker itself is binary—either allowing or blocking all traffic.

Can a specific model be locked out while its provider’s circuit remains closed?

Yes. The model-level lockout in lockoutPolicy.ts functions independently of the circuit breaker. A model can accumulate failures and become locked via markAccountUnavailable while the provider-wide circuit in circuitBreaker.ts remains CLOSED, allowing other models on the same provider to continue serving requests without interruption.

Where is resilience state persisted across restarts?

OmniRoute persists circuit-breaker state to SQLite via [src/lib/db/domainState.ts](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts) and model lockout records via [src/lib/db/lockoutState.ts](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/lockoutState.ts). This ensures that failure counts, cooldown timers, and lockout durations survive application restarts and maintain consistent protection across deployments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}