How OmniRoute Implements Its 3-Layer Resilience System: Circuit Breakers, Cooldown, and Lockout

OmniRoute protects LLM request routing with three coordinated resilience layers—circuit breakers at the provider level, cooldown at the connection level, and lockout at the model level—to prevent cascading failures and maintain service availability.

OmniRoute's 3-layer resilience system ensures robust request processing across multiple AI providers. By combining circuit breakers for provider-level protection, connection cooldown for credential-level isolation, and model lockout for fine-grained error handling, the system gracefully degrades only the problematic components while keeping healthy paths open. This implementation is live in the diegosouzapw/OmniRoute repository across TypeScript services in src/ and open-sse/ directories.

---

Architecture Overview: Three Resilience Layers

The resilience strategy operates at progressively narrower scopes, allowing precise failure containment:

Layer	Scope	Trigger	Recovery
Provider Circuit Breaker	Entire provider (e.g., openai, anthropic)	Repeated upstream failures	Timeout-based reset with HALF_OPEN probing
Connection Cooldown	Individual credential/account/key	Rate limits, transient errors	Exponential backoff per-connection
Model Lockout	Specific model on a connection	Model-specific quota errors	Manual or policy-based clearance

Each layer consults the next in sequence during request routing, creating a defense-in-depth pattern.

---

Layer 1: Provider Circuit Breaker

The circuit breaker stops all traffic to a provider that repeatedly fails at the upstream level, preventing a single unhealthy provider from throttling the entire service.

State Machine Implementation

In src/shared/utils/circuitBreaker.ts, the CircuitBreaker class maintains three states:

• CLOSED: Normal operation, requests allowed
• OPEN: Provider failing, requests rejected immediately
• HALF_OPEN: Probe request allowed to test recovery

// src/shared/utils/circuitBreaker.ts (excerpt)
const DEFAULTS = {
  oauth: { threshold: 3, resetTimeoutMs: 60_000 },
  apiKey: { threshold: 5, resetTimeoutMs: 30_000 },
};

The oauth threshold (3 failures) is stricter than apiKey (5 failures) because OAuth-based providers typically indicate more systemic issues when they fail.

Integration Points

The breaker is consulted at two critical execution points:

• open-sse/handlers/chatHelpers.ts – Validates provider health before request execution
• open-sse/services/accountFallback.ts – Checks canExecute() and getRetryAfterMs() during fallback decisions

Lazy Recovery Mechanism

OmniRoute uses lazy recovery: the breaker doesn't use timers to transition states. Instead, getStatus() automatically flips OPEN → HALF_OPEN when resetTimeoutMs has elapsed since the last failure. A single successful probe transitions to CLOSED; any failure reopens the circuit.

// Example: manually checking a provider before sending a request
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { Provider } from '@/shared/constants/providers';

async function safeExecute(provider: Provider, exec: () => Promise<any>) {
  const cb = getCircuitBreaker(provider);
  if (!cb.canExecute()) {
    throw new Error(`Provider ${provider} is circuit‑open`);
  }

  try {
    const result = await exec();
    cb.recordSuccess();        // resets failure count
    return result;
  } catch (err) {
    cb.recordFailure(err);     // may transition to OPEN
    throw err;
  }
}

---

Layer 2: Connection Cooldown

While the circuit breaker handles provider-wide failures, connection cooldown isolates individual bad credentials without disabling the entire provider. This allows other accounts or API keys for the same provider to continue serving traffic.

Cooldown State Management

In src/sse/services/auth.ts, the markAccountUnavailable() function implements exponential backoff:

// src/sse/services/auth.ts (excerpt)
function markAccountUnavailable(account, error) {
  const backoff = baseCooldownMs * 2 ** account.backoffLevel;
  account.rateLimitedUntil = new Date(Date.now() + backoff);
  account.backoffLevel++;
}

The backoff multiplier (2 ** failureIndex) ensures that repeatedly failing connections are sidelined for progressively longer periods, starting from baseCooldownMs.

Credential Selection Logic

The selectAvailableCredentials() function (also in auth.ts) filters out credentials where rateLimitedUntil is still in the future. This check is invoked by open-sse/services/accountFallback.ts::checkFallbackError() when evaluating fallback candidates.

// Example: using the connection selector that respects cooldowns
import { selectAvailableCredentials } from '@/sse/services/auth';

async function pickCredential(provider: string) {
  const creds = await selectAvailableCredentials(provider);
  if (!creds.length) {
    throw new Error(`All accounts for ${provider} are on cooldown`);
  }
  return creds[0]; // simple first‑available strategy
}

---

Layer 3: Model Lockout

The finest-grained layer, model lockout, isolates a single unavailable or quota-limited model while leaving other models on the same connection operational. This handles per-model rate limits common in LLM APIs (e.g., GPT-4 vs. GPT-3.5 on the same OpenAI organization).

Implementation in Account Fallback Service

In open-sse/services/accountFallback.ts, the checkFallbackError() function distinguishes between connection-level and model-specific errors:

// open-sse/services/accountFallback.ts (excerpt)
if (isTransientProviderError(err)) {
  // apply connection cooldown
  markAccountUnavailable(conn, err);
} else if (isModelSpecificError(err)) {
  // apply model lockout
  conn.modelLockout[modelId] = true;
}

Error Classification Logic

• HTTP 429 with model-specific indicator → Model lockout (e.g., quota exhausted for gpt-4-turbo only)
• HTTP 500-599 or network timeouts → Connection cooldown (affects the entire credential)

// Example: handling a model‑specific error to trigger lockout
import { handleModelError } from '@/open-sse/services/accountFallback';

function onModelResponse(err: any, modelId: string, conn: any) {
  if (err.code === 429 && err.isModelSpecific) {
    conn.modelLockout[modelId] = true; // lockout this model only
  } else {
    markAccountUnavailable(conn, err); // fallback to connection cooldown
  }
}

---

Request Flow: How Layers Interact

The open-sse/services/combo.ts router orchestrates all three resilience checks:

1. Request validation → Zod schema, optional auth, policy checks
2. Target resolution → resolveComboTargets() enumerates candidate providers
3. Circuit breaker check → handleSingleModel() calls circuitBreaker.canExecute(); OPEN providers are skipped
4. Credential filtering → Connection selector excludes rateLimitedUntil timestamps in the future
5. Execution & error handling → On failure, accountFallback.checkFallbackError() applies cooldown or lockout based on error classification
6. Retry with fallback → Router proceeds to next candidate if current fails

This sequential validation ensures that no request reaches a known-bad provider, credential, or model.

---

Configuration Constants

Default thresholds are centralized in open-sse/config/constants.ts:

Parameter	OAuth Value	API Key Value	Rationale
Failure threshold	3	5	OAuth failures indicate broader issues
Reset timeout	60,000 ms	30,000 ms	OAuth providers recover more slowly

These defaults can be overridden per-provider via environment configuration.

---

Summary

OmniRoute's 3-layer resilience system provides comprehensive failure isolation:

• Provider circuit breakers prevent cascading outages by blocking traffic to failing upstreams, with configurable thresholds and lazy recovery
• Connection cooldown implements exponential backoff for individual credentials, preserving provider capacity through alternative accounts
• Model lockout enables per-model isolation without connection disruption, critical for multi-model provider integrations

Together, these mechanisms ensure that OmniRoute routes around failures at the appropriate granularity—provider, credential, or model—maintaining service availability even during partial upstream outages.

---

Frequently Asked Questions

What triggers a circuit breaker to open in OmniRoute?

A circuit breaker opens when failure count exceeds the configured threshold (3 for OAuth, 5 for API keys) within a single operational window. Failures are recorded via recordFailure() in src/shared/utils/circuitBreaker.ts. The breaker stays open until resetTimeoutMs elapses, then transitions to HALF_OPEN on the next status check.

How does connection cooldown differ from circuit breaker state?

Circuit breaker operates at the provider level and uses fixed timeouts (30-60 seconds). Connection cooldown operates per-credential with exponential backoff starting from baseCooldownMs. A provider can have its circuit closed while individual connections are on cooldown, allowing other credentials for that provider to serve requests.

Can a model be locked out while its connection remains active?

Yes. Model lockout in open-sse/services/accountFallback.ts marks only conn.modelLockout[modelId] = true, leaving the connection and other models available. This commonly occurs when a specific model hits quota limits (HTTP 429) while the underlying credential remains valid for other models.

What happens when all resilience layers have excluded all candidates?

If no providers pass circuit breaker checks, no credentials escape cooldown, and no models avoid lockout, the router returns an aggregated error indicating exhaustion of all fallback paths. This is surfaced in open-sse/services/combo.ts when resolveComboTargets() yields an empty candidate set.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}