Understanding OmniRoute's Three Resilience Layers: Provider Breaker, Connection Cooldown, and Model Lockout

OmniRoute implements three specialized resilience layers—provider circuit breaker, connection cooldown, and model lockout—to isolate failures and maintain routing performance without disabling entire services.

The OmniRoute open-source routing proxy (by diegosouzapw/OmniRoute) protects LLM request pipelines through a tiered defense system. Each resilience layer targets a specific failure scope, enabling granular control over how the system responds to provider outages, rate limits, and model-specific errors.

The Three Resilience Layers Explained

OmniRoute's resilience architecture operates on three distinct scopes:

Layer	Scope	Primary Failure Mode
Provider Circuit Breaker	Entire provider (e.g., openai, anthropic)	Upstream service degradation
Connection Cooldown	Single connection/account/key	Rate limits, auth errors
Model Lockout	Provider + connection + model triple	Per-model quotas, missing models

This separation prevents over-reaction—a temporary model quota outage won't trigger a full provider shutdown.

Layer 1: Provider Circuit Breaker

The provider circuit breaker stops all traffic to a provider when systemic failures exceed thresholds, protecting downstream latency from cascading degradation.

Four-State State Machine

The breaker cycles through states defined in src/shared/utils/circuitBreaker.ts:

• CLOSED — Normal operation, requests pass through
• DEGRADED — Elevated errors, reduced but non-zero traffic allowed
• OPEN — Hard stop; all requests fast-fail for configured timeout
• HALF_OPEN — Probe request tests recovery after timeout expires

State transitions are lazy: the breaker only evaluates HALF_OPEN on the next status check after the OPEN timeout expires.

Core Implementation Files

• CircuitBreaker class: src/shared/utils/circuitBreaker.ts
• Request pipeline integration: src/sse/handlers/chatHelpers.ts, src/sse/handlers/chat.ts
• Monitoring endpoint: GET /api/monitoring/health in src/app/api/monitoring/health/route.ts
• Manual reset API: POST /api/resilience/reset in src/app/api/resilience/reset/route.ts

// Inspect provider breaker status programmatically
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('openai');
console.log(breaker.getStatus());
// => { name: 'openai', state: 'CLOSED', failureCount: 0, ... }

// Administrative reset of all breakers
import { resetAllCircuitBreakers } from '@/shared/utils/circuitBreaker';

resetAllCircuitBreakers(); // Clears breaker state and database rows

Layer 2: Connection Cooldown

When a specific API key hits rate limits or authentication errors, connection cooldown temporarily suspends that key while preserving other keys for the same provider.

How Connection Cooldown Works

The mechanism isolates failures to individual credentials without provider-wide impact. Key functions in src/sse/services/auth.ts handle marking and selection:

• markAccountUnavailable() — Places a connection into cooldown with configurable duration
• getProviderCredentials*() — Skips cooling-down connections during credential selection

The checkFallbackError() function in open-sse/services/accountFallback.ts evaluates whether errors trigger cooldown eligibility.

Configuration and Defaults

Cooldown behavior is configurable in src/lib/resilience/settings.ts:

• Base cooldown durations
• Exponential backoff multipliers
• Maximum backoff caps

// Manually trigger cooldown after rate limit detection
import { markAccountUnavailable, getProviderCredentials } from '@/sse/services/auth';

const credential = getProviderCredentials('openai', 'key-abc123');
if (credential) {
  markAccountUnavailable(credential, {
    rateLimitedUntil: Date.now() + 15_000, // 15 second cooldown
    testStatus: 'unavailable',
    backoffLevel: 1,
  });
}

Layer 3: Model Lockout

The most granular layer, model lockout, disables specific model + connection combinations. This prevents over-penalization when only one model in a multi-model connection experiences quota exhaustion or availability issues.

When Model Lockout Activates

Model lockout applies to scenarios including:

• Per-model API quotas (common in OpenAI tiered plans)
• Locally-hosted models that become unavailable
• Mode-specific restrictions (e.g., disabling vision models temporarily)

Implementation and APIs

Core functions in open-sse/services/accountFallback.ts:

• lockModel() — Creates a timed lockout entry
• clearModelLock() — Removes lockout manually or on expiration

Dashboard components provide operational visibility:

• ModelCooldownsCard.tsx — Read-only active lockout list
• ModelLockoutCard.tsx — Configuration and manual re-enable interface

REST endpoints at src/app/api/resilience/model-cooldowns/*:

• GET /api/resilience/model-cooldowns — List all active lockouts
• DELETE /api/resilience/model-cooldowns — Clear specific or all lockouts

// Lock a specific model after quota exhaustion
import { lockModel } from '@/open-sse/services/accountFallback';

await lockModel({
  provider: 'anthropic',
  connectionId: 'conn-def456',
  model: 'claude-2.0',
  reason: 'quota_exhausted',
  expiresAt: Date.now() + 120_000, // 2 minute lockout
});

Debugging Resilience Issues

When investigating routing failures, examine layers in order of scope:

1. Check provider breaker state — Query /api/monitoring/health or call getCircuitBreaker(provider).getStatus()
2. Inspect connection cooldowns — Review credential availability in src/sse/services/auth.ts logs
3. Verify model lockouts — Query GET /api/resilience/model-cooldowns for targeted restrictions

Each layer generates distinct telemetry, enabling precise root cause identification without disabling healthy infrastructure.

Summary

• Provider circuit breaker (src/shared/utils/circuitBreaker.ts) protects against systemic provider failures through a four-state machine with lazy recovery transitions
• Connection cooldown (src/sse/services/auth.ts) isolates individual API keys experiencing rate limits while maintaining provider availability
• Model lockout (open-sse/services/accountFallback.ts) enables per-model granularity for quota and availability management

These three resilience layers form OmniRoute's defense-in-depth strategy, ensuring graceful degradation without unnecessary service disruption.

Frequently Asked Questions

How do I manually reset a tripped provider circuit breaker?

Call resetAllCircuitBreakers() from src/shared/utils/circuitBreaker.ts or POST to /api/resilience/reset. This clears all breaker states and database persistence. Individual breaker reset is not exposed—design intent requires addressing the underlying provider issue before restoration.

What causes a connection to enter cooldown versus triggering the provider breaker?

Connection cooldown activates on credential-specific errors (401/429 on a single key) via markAccountUnavailable(). The provider breaker trips only when aggregated failures across all keys for that provider exceed thresholds, indicating upstream service degradation rather than isolated key issues.

Can model lockouts expire automatically or require manual clearing?

Both mechanisms exist. The lockModel() function accepts an expiresAt timestamp for automatic expiration. Alternatively, use the DELETE /api/resilience/model-cooldowns endpoint or dashboard ModelLockoutCard.tsx for immediate manual clearance before timeout.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}