OmniRoute's 3-Layer Resilience System: Circuit Breaker, Connection Cooldown, and Model Lockout

OmniRoute implements a three-layer resilience system that isolates failures at the provider, connection, and model levels to prevent cascading outages in AI routing pipelines.

OmniRoute is an open-source routing pipeline that protects AI model integrations through hierarchical fault tolerance. The OmniRoute's 3-layer resilience system combines circuit breakers, connection cooldowns, and model lockouts to handle upstream provider failures without manual intervention. This architecture ensures that transient errors like rate limits or quota exhaustion do not cascade through the entire service.

Layer 1: Provider-Wide Circuit Breaker

The first layer guards all requests routed through a specific provider using a state-machine-based circuit breaker implemented in src/shared/utils/circuitBreaker.ts.

State Machine and Thresholds

The circuit breaker tracks consecutive failures against a configurable failureThreshold (default 5). It transitions through four states:

• CLOSED: Normal operation; requests flow through
• DEGRADED: Early warning state before full isolation
• OPEN: Circuit tripped; all requests short-circuit immediately
• HALF_OPEN: Probe period allowing limited test requests (halfOpenRequests)

When failures exceed the threshold, the breaker moves from CLOSED to OPEN, blocking further traffic to that provider. After the resetTimeout expires, it enters HALF_OPEN to test recovery. Successful probes close the circuit; failures reopen it.

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,      // 30 seconds
  halfOpenRequests: 1
});

async function callProvider() {
  if (!breaker.canExecute()) {
    throw new Error('Provider unavailable (circuit open)');
  }
  return breaker.execute(async () => {
    // HTTP request logic here
  });
}

Layer 2: Connection Cooldown (Per-Failure-Kind Back-off)

The second layer implements connection cooldown—fine-grained back-off logic that applies different timeouts based on specific error types rather than treating all failures equally.

Cooldown by Failure Kind

Within the same circuitBreaker.ts file, the cooldownByKind map and classifyError logic allow each failure type (e.g., rate-limit, quota-exhausted, transient) to specify its own cooldown duration. The breaker selects the most restrictive timeout (_effectiveCooldown) for the last failure kind encountered.

This prevents "hammering" a provider when it returns HTTP 429 status codes or quota-exhausted errors. You can override the generic resetTimeout with specific durations for sensitive error types:

const breaker = getCircuitBreaker('anthropic', {
  resetTimeout: 30_000,
  cooldownByKind: {
    rate_limit: 120_000,      // 2 minutes for rate limits
    quota_exhausted: 300_000  // 5 minutes for quota issues
  }
});

Layer 3: Model-Level Lockout

The third layer provides granular protection at the individual model-connection level, implemented in src/domain/lockoutPolicy.ts and consumed by the account fallback service at src/services/accountFallback.ts.

Lockout Mechanics

Model lockout tracks failures per identifier using a sliding window (attemptWindowMs). When maxAttempts is exceeded within the window, a lockout record is persisted to the domain_lockout_state table with a configurable lockoutDurationMs.

While locked, the specific model-connection pair (e.g., Gemini-1.5-flash on account conn-123) is excluded from combo resolution. The lockout clears automatically after the duration expires, or immediately upon recordSuccess or explicit forceUnlock.

import { markAccountUnavailable } from '@/services/accountFallback';

// Lock out a specific model after repeated 429 errors
await markAccountUnavailable({
  provider: 'gemini',
  connectionId: 'conn-123',
  model: 'gemini-1.5-flash',
  reason: 'rate_limit'
});

// The lockout is automatically consulted during combo resolution
const combo = await resolveComboTargets(...); // Excludes locked models

How the Three Layers Interact

The OmniRoute's 3-layer resilience system operates hierarchically to minimize blast radius:

1. Circuit-breaker acts as the first gate. If the breaker is OPEN, the provider is excluded entirely from combo resolution.
2. Connection-cooldown applies when the breaker is CLOSED or DEGRADED, delaying the next request based on the specific failure kind via cooldownByKind.
3. Model-lockout provides the final safety net, isolating individual misbehaving models even when their parent provider is healthy.

This hierarchy ensures that systemic outages trigger provider-wide protection, while transient spikes affect only specific error types, and chronic model issues result in targeted lockouts.

Implementation Reference

Key Files and Functions

• src/shared/utils/circuitBreaker.ts: Core circuit breaker implementation with getCircuitBreaker(), canExecute(), and execute() methods
• src/domain/lockoutPolicy.ts: Model lockout logic with recordFailure(), recordSuccess(), and forceUnlock() functions
• src/services/accountFallback.ts: Routing logic that consults lockout state via markAccountUnavailable()
• src/lib/db/domainState.ts: SQLite persistence for circuit breaker states
• src/lib/db/lockoutState.ts: SQLite persistence for model lockout records

Monitoring Resilience State

To inspect current system health for dashboards or debugging:

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';
import { getAllModelLockouts } from '@/domain/lockoutPolicy';

console.log('Provider breakers:', getAllCircuitBreakerStatuses());
console.log('Model lockouts:', await getAllModelLockouts());

Summary

• Provider-Wide Circuit Breaker: Tracks consecutive failures (default threshold of 5) across a provider, transitioning through CLOSED → DEGRADED → OPEN → HALF_OPEN states to prevent cascading failures
• Connection Cooldown: Implements per-failure-kind back-off via cooldownByKind configuration, applying specific timeouts for rate limits versus quota exhaustion
• Model Lockout: Isolates individual model-connection pairs using sliding window counting (maxAttempts within attemptWindowMs) stored in domain_lockout_state
• Hierarchical Protection: Layers operate sequentially—circuit breakers protect providers, cooldowns throttle specific error types, and lockouts target individual models

Frequently Asked Questions

What triggers the circuit breaker to open?

The circuit breaker opens when consecutive failures exceed the failureThreshold parameter (default 5), as tracked in src/shared/utils/circuitBreaker.ts. Once opened, it remains in the OPEN state for the duration specified by resetTimeout before attempting recovery via the HALF_OPEN state.

How does connection cooldown differ from the circuit breaker?

While the circuit breaker operates at the provider level and uses a binary open/closed state, connection cooldown provides granular back-off per failure type. The cooldownByKind map in circuitBreaker.ts allows different timeouts for specific errors (e.g., 120 seconds for rate limits versus 30 seconds for generic timeouts), whereas the circuit breaker uses a single resetTimeout for all failures.

Can you manually unlock a model before the lockout expires?

Yes. According to the implementation in src/domain/lockoutPolicy.ts, you can clear a lockout immediately by calling recordSuccess() after a successful request, or by invoking forceUnlock() to explicitly remove the restriction. Otherwise, the lockout automatically expires after lockoutDurationMs.

How long do model lockouts typically last?

The duration is configurable per provider via lockoutDurationMs in the lockout policy configuration. The default values depend on the provider's reliability profile, but the system stores these durations in src/lib/db/lockoutState.ts and consults them during the combo resolution phase in src/services/accountFallback.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}