OmniRoute Resilience Layers: Circuit Breaker, Connection Cooldown, and Model Lockout Explained

OmniRoute implements a three-layer resilience system using provider-wide circuit breakers, per-failure-kind connection cooldowns, and model-level lockouts to isolate upstream failures and prevent cascading outages.

OmniRoute is an open-source AI routing layer designed to maintain service continuity during upstream provider instability. The codebase in diegosouzapw/OmniRoute implements a hierarchical defense mechanism that isolates failures at the provider, connection, and model levels. These resilience layers ensure that transient errors or rate limits from one upstream source do not degrade the entire routing pipeline.

The Three Resilience Layers

OmniRoute protects the routing pipeline through a granular, hierarchical system that operates at increasing levels of specificity.

Provider-Wide Circuit Breaker

The first layer guards all requests targeting a specific provider using a state-machine-based circuit breaker implemented in src/shared/utils/circuitBreaker.ts. This component tracks consecutive failures across any model on that provider and implements a CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED state transition.

When failures exceed the configurable failureThreshold (default 5), the breaker transitions to OPEN, short-circuiting further calls to that provider. After the resetTimeout period expires, the state shifts to HALF_OPEN, allowing a limited number of probe requests (halfOpenRequests). Successful probes close the circuit, while failed probes reopen it.

The circuit breaker exposes canExecute() to check availability and execute() to wrap upstream calls with automatic failure tracking and state management.

Connection Cooldown (Per-Failure-Kind Back-off)

The second layer adds per-failure-type granularity within the same circuitBreaker.ts file. Instead of applying a uniform cooldown, OmniRoute uses a cooldownByKind map and classifyError logic to assign specific back-off periods to different error categories such as rate-limit, quota-exhausted, or transient failures.

When a failure occurs, the breaker calculates an _effectiveCooldown by selecting the most restrictive timeout between the generic resetTimeout and the specific cooldown for that failure kind. This prevents "hammering" a provider with repeated requests when it returns HTTP 429 or quota-exhausted errors, allowing aggressive retries for transient network errors while enforcing longer delays for rate limits.

Model-Level Lockout

The third layer operates at the individual model-connection pair level, isolating specific misbehaving endpoints without disabling the entire provider. Implemented in src/domain/lockoutPolicy.ts and consumed by the account-fallback service in src/services/accountFallback.ts, this layer tracks failures per identifier using a sliding window approach.

The system counts failures up to maxAttempts within an attemptWindowMs period. When the threshold is crossed, a lockout record is persisted to the domain_lockout_state table with a lockoutDurationMs expiration. While locked, the model is excluded from combo resolution and selection. A successful request via recordSuccess() or an explicit forceUnlock clears the lockout immediately.

How the Layers Interact

These resilience mechanisms operate hierarchically to provide defense in depth:

1. Circuit breaker determines if a provider is usable at all. If the breaker is OPEN, the provider is excluded entirely from combo resolution.
2. When the circuit is CLOSED or DEGRADED, the connection cooldown layer may still delay the next request based on the specific failure kind (cooldownByKind), applying the calculated _effectiveCooldown before allowing traffic.
3. Even when a provider is healthy, the model lockout layer can temporarily drop a specific model-connection pair (e.g., Gemini-1.5-flash on account conn-123) that repeatedly returns 429 or 404 errors, consulting the domain_lockout_state table during the combo selection process.

This architecture ensures that systemic outages trigger provider-wide isolation, rate-limit spikes trigger connection-specific delays, and individual model failures trigger targeted lockouts without affecting healthy upstream capacity.

Implementation Examples

Using the Circuit Breaker for Upstream Calls

The following pattern demonstrates how to wrap provider calls with circuit breaker protection:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Create or retrieve a breaker for the "openai" provider
const breaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,      // 30s default cooldown
  halfOpenRequests: 1,
  // Optional per-kind overrides (e.g., 429 → 2 min)
  cooldownByKind: { rate_limit: 120_000 },
});

async function callOpenAI(payload: any) {
  // Check if the circuit allows execution
  if (!breaker.canExecute()) {
    throw new Error('OpenAI is currently unavailable (circuit open)');
  }

  // Wrap the actual HTTP call
  return breaker.execute(async () => {
    // ... fetch() implementation
  });
}

Applying Model-Level Lockout

When a specific model returns a 429 error, trigger a lockout via the account fallback service:

import { markAccountUnavailable } from '@/services/accountFallback';

// Mark a specific model-connection as unavailable
await markAccountUnavailable({
  provider: 'gemini',
  connectionId: 'conn-123',
  model: 'gemini-1.5-flash',
  reason: 'rate_limit',
  // Lockout persists for the provider's default duration
});

// The combo resolver will automatically skip this locked model
const combo = await resolveComboTargets(...);

Monitoring Resilience State

Inspect current system health using exported status functions:

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';
import { getAllModelLockouts } from '@/domain/lockoutPolicy';

console.log('Provider breakers:', getAllCircuitBreakerStatuses());
console.log('Model lockouts:', await getAllModelLockouts());

Summary

• Provider-wide circuit breakers in src/shared/utils/circuitBreaker.ts isolate entire providers after 5 consecutive failures, using a state machine with configurable resetTimeout and halfOpenRequests.
• Connection cooldowns provide per-failure-type back-off via cooldownByKind, allowing different delays for rate limits versus transient errors.
• Model-level lockouts in src/domain/lockoutPolicy.ts track failures per model-connection pair using sliding windows (maxAttempts within attemptWindowMs) and persist blocks to domain_lockout_state.
• The account-fallback service (src/services/accountFallback.ts) consumes lockout data to exclude degraded models from routing decisions.
• These layers operate hierarchically: circuit breakers exclude unhealthy providers, cooldowns delay specific error types, and lockouts isolate individual models.

Frequently Asked Questions

How does the circuit breaker state machine transition between states?

The circuit breaker starts in CLOSED (normal operation). After failureThreshold consecutive failures (default 5), it transitions to OPEN, rejecting all requests. After resetTimeout milliseconds, it enters HALF_OPEN, allowing halfOpenRequests probe calls. Successful probes transition back to CLOSED, while failed probes return to OPEN. An intermediate DEGRADED state may also indicate elevated error rates before full opening.

What is the difference between connection cooldown and the circuit breaker?

The circuit breaker is binary—it either allows all traffic or blocks all traffic to a provider based on aggregate failure counts. Connection cooldown operates within an open circuit or during degraded states, applying specific delays based on the type of error encountered (e.g., 120 seconds for rate limits vs. 30 seconds for generic errors). This prevents hammering providers with 429 errors while allowing faster retries for transient network blips.

How long does a model lockout persist?

Model lockouts persist for the duration specified by lockoutDurationMs in the lockout policy, which varies by provider configuration. The lockout is stored in the domain_lockout_state table and automatically expires after the duration elapses. Alternatively, a successful request to that model-connection pair via recordSuccess() or an explicit forceUnlock call will immediately clear the lockout.

Can resilience thresholds be customized per provider?

Yes. When retrieving a circuit breaker via getCircuitBreaker(), you can specify custom failureThreshold, resetTimeout, halfOpenRequests, and a cooldownByKind map to override defaults for specific error types. Similarly, model lockout parameters including maxAttempts, attemptWindowMs, and lockoutDurationMs are configurable per provider in the domain policy configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}