How OmniRoute Handles Circuit Breakers and Resilience: A Complete Technical Guide

OmniRoute implements a three-layer resilience architecture—provider circuit breakers, connection cooldowns, and model lockouts—with lazy recovery and database persistence to isolate failures at progressively finer scopes.

OmniRoute is an open-source AI model routing proxy that combines multiple LLM providers behind a unified API. Its circuit breaker and resilience system prevents cascading failures when upstream providers (OpenAI, Anthropic, etc.) experience outages, rate limits, or quota exhaustion. The implementation spans src/shared/utils/circuitBreaker.ts, src/sse/services/auth.ts, and multiple supporting modules that work together to maintain service availability.

Three-Layer Resilience Architecture

OmniRoute organizes failure isolation into three distinct scopes, each with dedicated implementation files and state management strategies.

Layer	Scope	Core File	Purpose
Provider Circuit Breaker	Entire provider (e.g., openai, anthropic)	src/shared/utils/circuitBreaker.ts	Blocks traffic to failing providers after repeated 5xx errors
Connection Cooldown	Individual API key / credential	src/sse/services/auth.ts	Temporarily excludes bad credentials while keeping other keys active
Model Lockout	Provider + connection + model triple	open-sse/services/accountFallback.ts	Prevents a single model from blocking an otherwise healthy connection

All three layers share a lazy-recovery philosophy: state refreshes on demand during status checks rather than using background timers, reducing CPU overhead and complexity.

Provider Circuit Breaker Implementation

The circuit breaker in src/shared/utils/circuitBreaker.ts provides the outermost protection layer. It tracks failure rates per provider and transitions through four states to balance availability against error propagation.

State Machine Design

The breaker implements CLOSED → DEGRADED → OPEN → HALF_OPEN transitions:

• CLOSED: Normal operation; requests pass through
• DEGRADED: Elevated failure rate detected; requests still allowed but monitored
• OPEN: Provider blocked; all requests fail fast with retryAfterMs guidance
• HALF_OPEN: Probe mode; limited test requests allowed to verify recovery

// src/shared/utils/circuitBreaker.ts – core state transition logic
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const openAiBreaker = getCircuitBreaker('openai', {
  failureThreshold: 8,           // Transition to OPEN after 8 failures
  resetTimeout: 30_000,          // 30s base cooldown
  halfOpenRequests: 2,           // Allow 2 probe requests in HALF_OPEN
  maxBackoffMultiplier: 4,       // Cap exponential backoff at 4x
  
  // Classify errors: only upstream 5xx triggers breaker
  isFailure: (err) => err?.statusCode >= 500,
});

Adaptive Back-Off and Persistence

The _effectiveResetTimeout() method implements exponential back-off: each OPEN → HALF_OPEN → OPEN cycle multiplies the reset timeout up to maxBackoffMultiplier. Breaker state persists to the domain_circuit_breakers table via saveCircuitBreakerState() and loadCircuitBreakerState(), ensuring recovery after process restarts.

A global Map<string, CircuitBreaker> registry caps at 500 entries, with idle CLOSED breakers automatically evicted via evictColdBreakersIfNeeded().

Integration Points

The breaker wires into the request pipeline through src/sse/handlers/chatHelpers.ts and src/sse/handlers/chat.ts. Operational visibility comes from:

• GET /api/monitoring/health – exposes breaker states
• POST /api/resilience/reset – manual reset endpoint

// Wrapping an upstream request with circuit breaker protection
async function fetchOpenAIChat(payload: any) {
  return openAiBreaker.execute(async () => {
    const resp = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
      body: JSON.stringify(payload),
    });
    if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
    return resp.json();
  });
}

// Health check integration
const status = openAiBreaker.getStatus();
console.log('State:', status.state, 'Retry after:', status.retryAfterMs);

Connection Cooldown: Per-Credential Isolation

When a specific API key hits rate limits or network errors, connection cooldown isolates that credential without affecting other keys for the same provider.

Implementation in auth.ts

The markAccountUnavailable() function in src/sse/services/auth.ts records failure metadata:

• rateLimitedUntil: Timestamp when the connection becomes eligible again
• testStatus: Current verification state of the credential
• backoffLevel: Exponential backoff iteration count

checkFallbackError() in open-sse/services/accountFallback.ts evaluates these fields during request routing. The cooldown uses lazy expiration: connections are excluded while rateLimitedUntil is in the future, then automatically become eligible without timer cleanup.

Back-Off Configuration

Default cooldown bases are defined in src/lib/resilience/settings.ts:

• OAuth keys: 5 seconds base
• API keys: 3 seconds base

Cooldown duration calculates as baseCooldownMs × 2ⁿ where n is the failure index.

// Simplified connection cooldown usage
import { markAccountUnavailable } from '@/sse/services/auth';

async function handleRequest(connId: string, fn: () => Promise<any>) {
  try {
    return await fn();
  } catch (err) {
    if (isRateLimitOrNetworkError(err)) {
      await markAccountUnavailable(connId, err); // Triggers exponential backoff
    }
    throw err;
  }
}

Model Lockout: Fine-Grained Quota Protection

The innermost resilience layer prevents individual models from degrading connection health. A model returning 404 (unknown model) or 429 (per-model quota) triggers lockout for the specific provider:connectionId:model triple.

Lockout Mechanics

lockModel() and clearModelLock() in open-sse/services/accountFallback.ts manage an in-memory Map of lockout entries. Configuration lives in src/lib/resilience/modelLockoutSettings.ts:

// DEFAULT_MODEL_LOCKOUT_SETTINGS structure
{
  enabled: boolean,
  errorCodes: number[],        // e.g., [404, 429]
  baseCooldownMs: number,
  maxBackoffMultiplier: number,
  failureThreshold: number,    // Lock after N failures
}

Recovery via Decay

Unlike the provider breaker, model lockout includes failure count decay: successful calls increment a decay counter (decayModelFailureCount), allowing models to recover before the lockout timer expires. This prevents permanent exclusion of temporarily strained models.

Users can view and clear lockouts via Settings → Model Lockouts in the UI, or programmatically through the API.

// Model lockout trigger on model-specific errors
import { lockModel, clearModelLock } from '@/open-sse/services/accountFallback';

async function requestModel(provider: string, connId: string, model: string, fn: () => Promise<any>) {
  try {
    return await fn();
  } catch (err) {
    if (lockoutSettings.errorCodes.includes(err.statusCode)) {
      await lockModel(provider, connId, model, err);
    }
    throw err;
  }
}

// Manual recovery
await clearModelLock(provider, connId, model);

Per-Failure-Kind Thresholds

The provider circuit breaker supports granular failure classification through kindThresholds and cooldownByKind. Failures are categorized (e.g., rate_limit, quota_exhausted, transient) with distinct:

• Threshold counts for state transitions
• Immediate-open flags for critical error types
• Custom cooldown durations per kind

This prevents, for example, rate-limit 429s from opening the breaker (handled by connection cooldown instead) while treating 503 service-unavailable errors as breaker-triggering.

Key Source Files and References

Component	File Path	Purpose
Circuit breaker core	src/shared/utils/circuitBreaker.ts	State machine, persistence, registry
Resilience documentation	docs/architecture/RESILIENCE_GUIDE.md	Architecture overview and configuration
Chat handler integration	src/sse/handlers/chatHelpers.ts	Request pipeline wiring
Connection cooldown	src/sse/services/auth.ts	markAccountUnavailable() implementation
Model lockout logic	open-sse/services/accountFallback.ts	lockModel(), clearModelLock()
Model lockout settings	src/lib/resilience/modelLockoutSettings.ts	DEFAULT_MODEL_LOCKOUT_SETTINGS
Health API	src/app/api/monitoring/health/route.ts	Breaker state exposure
Reset API	src/app/api/resilience/reset/route.ts	Manual breaker reset endpoint

Summary

OmniRoute's circuit breaker and resilience system provides defense in depth through three coordinated mechanisms:

• Provider circuit breakers (src/shared/utils/circuitBreaker.ts) stop traffic to failing providers with a four-state machine, adaptive exponential back-off, and database persistence
• Connection cooldowns (src/sse/services/auth.ts) isolate bad API keys using lazy timestamp-based exclusion without background timers
• Model lockouts (open-sse/services/accountFallback.ts) prevent individual models from blocking healthy connections, with decay-based recovery

All layers implement lazy recovery—state evaluation happens on demand during routing decisions, eliminating timer management overhead. The system exposes operational visibility through health endpoints and supports manual intervention via reset APIs and UI controls.

Frequently Asked Questions

What happens when a provider circuit breaker opens?

When the breaker opens, all new requests to that provider fail fast with a retryAfterMs value indicating when to retry. The breaker enters HALF_OPEN state after the reset timeout, allows a limited number of probe requests, and transitions back to CLOSED if they succeed or OPEN with increased back-off if they fail.

How does OmniRoute distinguish between provider failures and model-specific errors?

The system uses error classification and scope-based handling. Provider-level 5xx errors trigger the circuit breaker in circuitBreaker.ts. Connection-level 429s trigger cooldown in auth.ts. Model-specific 404s or per-model 429s trigger lockout in accountFallback.ts. The isFailure predicate and kindThresholds configuration control this routing.

Can circuit breaker state survive process restarts?

Yes. The saveCircuitBreakerState() and loadCircuitBreakerState() functions persist breaker state to the domain_circuit_breakers database table. This ensures that accumulated failure history and back-off levels are not lost when the OmniRoute process restarts.

What's the difference between DEGRADED and OPEN states?

DEGRADED is a warning state where elevated failure rates are detected but requests still flow through; it allows quick recovery without full blocking. OPEN is a hard stop where all requests fail immediately. The DEGRADED state provides early indication before escalating to OPEN, reducing unnecessary traffic disruption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}