Resilience Layers in OmniRoute: Circuit Breakers, Connection Cooldown, and Model Lockout

OmniRoute implements a three-layer resilience system—provider-wide circuit breakers, per-failure-kind connection cooldown, and model-level lockout—to isolate flaky upstream providers and prevent cascading failures across the routing pipeline.

The diegosouzapw/OmniRoute repository protects its AI model routing infrastructure through hierarchical fault tolerance mechanisms. These resilience layers guard against systemic outages, rate-limit spikes, and individual model malfunctions without requiring manual intervention. This article examines the implementation details, configuration options, and interaction patterns between these safeguards.

The Three-Layer Resilience Architecture

OmniRoute's resilience strategy operates at three distinct granularities: the entire provider, specific failure types, and individual model connections.

Provider-Wide Circuit Breaker

The first layer sits at src/shared/utils/circuitBreaker.ts, where the CircuitBreaker class monitors all requests targeting a specific provider (such as OpenAI or Gemini).

State Machine Implementation

The breaker maintains a five-state lifecycle: CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED.

• In the CLOSED state, requests flow normally while the system tracks consecutive failures
• After exceeding the failureThreshold (default 5), the breaker transitions to OPEN, short-circuiting all subsequent calls
• Following the resetTimeout period, the state shifts to HALF_OPEN, allowing a limited number of probe requests (halfOpenRequests)
• Successful probes close the circuit; failed probes reopen it

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Configure a breaker for the "openai" provider
const breaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,      // 30 seconds default cooldown
  halfOpenRequests: 1,
});

async function callOpenAI(payload: unknown) {
  if (!breaker.canExecute()) {
    throw new Error('OpenAI is currently unavailable (circuit open)');
  }

  return breaker.execute(async () => {
    // HTTP request logic here
    return await fetchProvider(payload);
  });
}

Connection Cooldown (Per-Failure-Kind Back-Off)

The second layer addresses specific failure classifications through the cooldownByKind map within the same circuitBreaker.ts file. This mechanism overrides the generic resetTimeout when the provider returns specific error types like rate limits or quota exhaustion.

Failure Classification Logic

The classifyError function categorizes failures, allowing the breaker to apply targeted back-off strategies. When a 429 (rate limit) or quota-exhausted error occurs, the system uses the _effectiveCooldown calculation to select the most restrictive timeout available for that specific failure kind.

const breaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,
  // Override cooldowns for specific failure types
  cooldownByKind: { 
    rate_limit: 120_000,      // 2 minutes for rate limits
    quota_exhausted: 300_000  // 5 minutes for quota issues
  },
});

This prevents "hammering" a provider during transient rejection periods while allowing faster recovery for other error types.

Model-Level Lockout

The third layer isolates individual model-connection pairs through src/domain/lockoutPolicy.ts. While the circuit breaker handles provider-wide health, the lockout policy tracks specific model instances (such as Gemini-1.5-flash on a particular account).

Lockout Mechanism

The LockoutPolicy class counts failures per identifier using a sliding window (attemptWindowMs). When failures exceed maxAttempts within this window, the system persists a lockout record for lockoutDurationMs. During lockout, the model is excluded from combo resolution and selection algorithms.

The src/services/accountFallback.ts service consumes this policy through markAccountUnavailable(), which triggers when specific models return repeated 429 or 404 errors.

import { markAccountUnavailable } from '@/services/accountFallback';

// Trigger lockout for a specific model returning rate limits
await markAccountUnavailable({
  provider: 'gemini',
  connectionId: 'conn-123',
  model: 'gemini-1.5-flash',
  reason: 'rate_limit',
});

// The combo resolver automatically skips locked models
const combo = await resolveComboTargets(...);

How the Layers Interact

These resilience mechanisms operate hierarchically during request routing:

1. Circuit-Breaker Evaluation: The system first checks if the provider's circuit breaker is OPEN. If so, the provider is excluded from combo resolution entirely.

2. Cooldown Application: When the breaker is CLOSED or DEGRADED, the connection cooldown layer may still apply delays based on the last failure kind detected. The cooldownByKind configuration determines the back-off duration before the next request attempt.

3. Lockout Verification: Even with a healthy provider circuit, the model-level lockout layer can temporarily blacklist specific model-connection pairs. The domain_lockout_state table persists these records, consulted by the account-fallback logic in src/services/accountFallback.ts.

4. Success Recovery: A successful request through recordSuccess() or an explicit forceUnlock clears the model lockout, while successful probe requests in HALF_OPEN state restore the circuit breaker to CLOSED.

Monitoring Resilience State

OmniRoute exposes the current state of all resilience layers for observability and dashboarding:

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';
import { getAllModelLockouts } from '@/domain/lockoutPolicy';

// Current circuit states for all providers
console.log('Provider breakers:', getAllCircuitBreakerStatuses());

// Active model lockouts
console.log('Model lockouts:', await getAllModelLockouts());

These helper functions, located at the end of circuitBreaker.ts and exported from lockoutPolicy.ts, enable real-time monitoring of which providers and models are currently isolated from the routing pool.

Summary

• Provider-Wide Circuit Breaker: Located in src/shared/utils/circuitBreaker.ts, implements a five-state machine (CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED) with configurable failureThreshold and resetTimeout to protect against systemic provider outages.

• Connection Cooldown: Implemented within the same circuit breaker file, uses the cooldownByKind map to apply specific back-off durations for different failure types (rate limits, quota exhaustion), preventing request hammering during transient errors.

• Model-Level Lockout: Defined in src/domain/lockoutPolicy.ts and consumed by src/services/accountFallback.ts, tracks failures per model-connection pair using sliding windows (attemptWindowMs) and temporarily excludes repeat offenders via persistent lockout records.

Frequently Asked Questions

How does OmniRoute determine when to open a circuit breaker?

The circuit breaker opens when consecutive failures exceed the failureThreshold parameter (default 5). According to the implementation in src/shared/utils/circuitBreaker.ts, the breaker tracks failure counts through the recordFailure() method. When the threshold is crossed, the state transitions from CLOSED or DEGRADED to OPEN, immediately short-circuiting further requests to that provider until the resetTimeout expires.

What is the difference between connection cooldown and model lockout?

Connection cooldown operates at the failure-type level within the circuit breaker, applying specific back-off durations based on error classification (such as 429 rate limits versus 500 errors). Model lockout operates at the individual model-connection level through src/domain/lockoutPolicy.ts, completely excluding specific model instances from routing after repeated failures. Cooldown delays requests to a provider, while lockout removes specific models from the candidate pool.

Can circuit breaker state persist across application restarts?

Yes, the circuit breaker implementation in OmniRoute supports persistence through src/lib/db/domainState.ts, which stores breaker state in SQLite. Similarly, model lockouts persist via src/lib/db/lockoutState.ts. This ensures that resilience decisions survive application restarts, preventing immediate re-quota-exhausted providers or re-locked models from receiving traffic before the system stabilizes.

How does the half-open state prevent thundering herd problems?

The HALF_OPEN state limits probe traffic through the halfOpenRequests parameter (typically set to 1). When the resetTimeout expires, the breaker allows only this specific number of test requests to pass through. Successful probes restore normal operation (CLOSED state), while failed probes immediately reopen the circuit. This prevents all waiting requests from simultaneously hitting a recovering provider.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}