OmniRoute Built-in Resilience Features: A 3-Layer Protection System Explained

OmniRoute protects request processing from flaky upstream providers through a provider-level circuit breaker, connection-level cooldowns, and model-level lockouts, plus additional runtime guards for quota control, queue admission, stream health, and status correction.

The OmniRoute proxy—maintained by diegosouzapw/OmniRoute—implements a sophisticated three-layer resilience stack that isolates failures at different granularities. Each layer operates independently, ensuring that a problem with one provider, credential, or model does not cascade into system-wide degradation. Understanding these built-in resilience features of OmniRoute helps operators configure effective failure boundaries and debug issues faster.

Provider Circuit Breaker: Stopping Catastrophic Outages

The provider circuit breaker in src/shared/utils/circuitBreaker.ts halts traffic to entire upstream providers (e.g., openai, anthropic) when they return repeated 5xx errors.

State Machine and Transitions

The breaker implements a five-state lifecycle:

• CLOSED → normal operation
• DEGRADED → elevated failure rate threshold reached
• OPEN → circuit tripped, requests blocked
• HALF_OPEN → probe request allowed after timeout
• CLOSED → recovery confirmed

Lazy recovery triggers automatically: when OPEN timeout expires, calls to getStatus(), canExecute(), or getRetryAfterMs() transition the breaker to HALF_OPEN and admit a single probe request.

Trigger Conditions and Scope

Only upstream 5xx errors (408, 500, 502, 503, 504) open the breaker. Credential-level failures (401, 403, 429) bypass this layer and are handled by connection cooldowns instead.

// Core implementation – src/shared/utils/circuitBreaker.ts
export class CircuitBreaker {
  private state: BreakerState = BreakerState.CLOSED;
  private failureCount = 0;
  private lastFailureTime?: number;

  // State transitions (lines 70-84, 110-135)
  // Lazy recovery via getStatus() / canExecute() / getRetryAfterMs()
}

Global Registry Management

The getCircuitBreaker(name) function lazily initializes breakers and evicts cold, closed entries when the registry exceeds MAX_REGISTRY_SIZE (lines 45-52). This prevents memory leaks in long-running deployments.

Connection Cooldown: Isolating Bad Credentials

While the circuit breaker protects at provider scope, connection cooldown targets individual credentials. When a specific API key hits rate limits or transient failures, OmniRoute marks it unavailable without affecting other keys for the same provider.

Implementation in auth.ts

The markAccountUnavailable function in src/sse/services/auth.ts updates provider_connections with cooldown metadata:

// Mark connection unavailable – src/sse/services/auth.ts
export async function markAccountUnavailable(
  connectionId: string,
  errorContext: ErrorContext
): Promise<void> {
  const cooldownMs = calculateBackoff(failureIndex);
  await updateProviderConnection(connectionId, {
    rateLimitedUntil: new Date(Date.now() + cooldownMs).toISOString(),
    testStatus: "unavailable",
    lastError: errorContext.message,
    backoffLevel: failureIndex,
  });
}

The selector checks new Date(rateLimitedUntil).getTime() > Date.now() when choosing connections (lines 64-70).

Exponential Back-off with Thundering Herd Protection

Cooldown duration follows baseCooldownMs * 2 ** failureIndex. A synchronization guard prevents concurrent failures from double-incrementing backoffLevel, ensuring predictable recovery times even under high concurrency.

Fallback Integration

The checkFallbackError function in open-sse/services/accountFallback.ts coordinates cooldown triggers with the broader fallback system, ensuring rapid failover to healthy connections.

Model Lockout: Granular Quota Isolation

Model lockout provides the finest-grained failure isolation: when a specific model (e.g., gpt-4-turbo) exhausts quota on a connection, other models on that same credential remain operational.

Scope and Keying

Lockouts are keyed by ${provider}:${connectionId}:${model} triple. This prevents over-quota scenarios on one model from disabling an entire connection with multiple model access.

Implementation in accountFallback.ts

// Lock a failing model – open-sse/services/accountFallback.ts
function lockModel(connectionId: string, model: string, reason: string): void {
  const key = `${provider}:${connectionId}:${model}`;
  const record = modelLockoutMap.get(key) || { failureCount: 0, lockedAt: null };
  record.failureCount++;
  if (record.failureCount >= MODEL_LOCKOUT_THRESHOLD) {
    record.lockedAt = Date.now();
  }
  modelLockoutMap.set(key, record);
}

Success-Decay Recovery

Unlike time-based expiration alone, successful requests automatically halve the stored failureCount. This allows organic recovery before the hard lockout timer expires, reducing false positives for intermittently flaky models.

Quota-Share Concurrency Control

Shared accounts—common with providers like GLM and MiniMax—require serialized access to prevent parallel requests from exhausting quota instantly.

Configuration and Runtime

Operators set provider_connections.max_concurrent via Dashboard → Settings → Resilience. At runtime, a per-connection semaphore (qsconn:<connectionId>) enforces the limit:

• Positive max_concurrent: requests queue and execute serially
• Zero or null: no concurrency restriction applies

The implementation spans src/lib/resilience/settings.ts for configuration and open-sse/services/combo.ts for semaphore acquisition.

Request Queue Admission Control

OmniRoute rejects overload before expensive work begins. The admission control layer in open-sse/services/rateLimitManager/admission.ts monitors Bottleneck queue depth against resilienceSettings.requestQueue.maxQueueDepth.

Fast-Fail Behavior

When queueLength >= maxQueueDepth, the system throws immediately:

// Admission check – open-sse/services/rateLimitManager/admission.ts
if (queueLength >= maxQueueDepth) {
  throw new RateLimitError(
    "Request queue at capacity",
    "RATE_LIMIT_QUEUE_FULL"
  );
}

This occurs before translation, compression, or upstream connection attempts—conserving resources during traffic spikes. Default maxQueueDepth is 0 (disabled); maxWaitMs defaults to 15000ms.

Slow-Stream Throughput Watchdog

Streaming responses can stall indefinitely without closing the connection. The throughput watchdog in open-sse/services/streamRecovery.ts monitors byte rate after a configurable warm-up period.

Activation and Measurement

Controlled by resilienceSettings.streamRecovery.throughputWatchdog (default disabled). Post warm-up, measured delta throughput below the configured minimum triggers stream abortion, freeing resources and allowing fallback to alternative providers.

Upstream Status Restatement

Some gateways misreport temporary quota exhaustion as 403 Forbidden instead of 429 Too Many Requests. OmniRoute corrects these before resilience classification.

Rule-Based Rewriting

open-sse/config/upstreamStatusRestatement.ts defines pattern-matching rules that rewrite matched responses to 429. This occurs in chatCore.ts before fallback logic executes, ensuring cooldown and lockout paths handle the error correctly.

Debugging and Operational Hooks

OmniRoute exposes resilience state for monitoring and manual intervention:

Endpoint/Purpose	Location
Health check all breakers	GET /api/monitoring/health in src/sse/handlers/chatHelpers.ts
Reset all breaker states	POST /api/resilience/reset → resetAllCircuitBreakers() in circuitBreaker.ts
Dashboard configuration	Settings panels for all three core mechanisms

Query Circuit Breaker Health Programmatically

import { getAllCircuitBreakerStatuses } from "@/shared/utils/circuitBreaker";

async function auditResilience(): Promise<void> {
  const statuses = await getAllCircuitBreakerStatuses();
  console.table(statuses.map(s => ({
    provider: s.name,
    state: s.state,
    failures: s.failureCount,
    retryAfter: s.retryAfterMs,
  })));
}

Source: getAllCircuitBreakerStatuses() at lines 621-627 in circuitBreaker.ts.

Manual Breaker Reset

import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";

async function resetProvider(name: string): Promise<void> {
  const breaker = getCircuitBreaker(name);
  breaker.reset(); // Clears state and persistent DB entry
}

Source: reset() method at lines 401-410.

Force Connection Cooldown After 429

import { markAccountUnavailable } from "@/sse/services/auth";

async function applyCooldown(
  connId: string,
  retryAfterSec: number
): Promise<void> {
  await markAccountUnavailable(connId, {
    message: "Manual cooldown after 429",
    retryAfter: retryAfterSec,
  });
}

Source: markAccountUnavailable at lines 380-390 in auth.ts.

Programmatic Model Lockout

import { lockModel } from "@/sse/services/accountFallback";

function isolateFailingModel(connId: string, model: string): void {
  lockModel(connId, model, "operator_triggered_lockout");
}

Source: lockModel in open-sse/services/accountFallback.ts.

Summary

OmniRoute's built-in resilience features operate through composable, layered mechanisms:

• Provider circuit breaker (src/shared/utils/circuitBreaker.ts) stops traffic to failing upstreams at the 5xx level
• Connection cooldown (src/sse/services/auth.ts) isolates individual credentials with exponential back-off
• Model lockout (open-sse/services/accountFallback.ts) preserves connection usability when only specific models fail
• Quota-share concurrency control serializes access to shared accounts
• Queue admission control (open-sse/services/rateLimitManager/admission.ts) rejects overload before expensive processing
• Slow-stream watchdog (open-sse/services/streamRecovery.ts) aborts stalled SSE connections
• Status restatement (open-sse/config/upstreamStatusRestatement.ts) corrects misreported errors for proper classification

All layers are configurable via environment, settings files, or the Dashboard UI, with programmatic hooks for health monitoring and manual state reset.

Frequently Asked Questions

What triggers an OmniRoute circuit breaker to open?

Only upstream 5xx HTTP status codes (408, 500, 502, 503, 504) increment the failure counter toward OPEN state. Client errors like 401, 403, and 429 are intentionally excluded—these are handled by connection cooldown and model lockout layers instead, preserving granular failure isolation.

How does OmniRoute handle rate limits differently from circuit breaker trips?

Rate limits (429) trigger connection cooldown, not circuit breaker opening. A single API key enters exponential back-off while other keys for the same provider remain active. This per-credential isolation prevents one exhausted quota from disabling an entire provider endpoint.

Can operators manually reset circuit breaker states?

Yes. The POST /api/resilience/reset endpoint clears all breaker states, or use getCircuitBreaker(name).reset() programmatically to target a specific provider. This is useful after confirmed upstream recovery or during incident response drills.

What happens when a model-specific quota is exhausted?

OmniRoute applies model lockout for the provider:connection:model triple. Other models on the same credential continue serving requests. The lockout includes success-decay recovery—successful requests on other models reduce the failure count organically before the timer expires.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}