How OmniRoute's Circuit Breaker, Connection Cooldown, and Model Lockout Interact

OmniRoute protects request routing through three complementary resilience layers: provider-level circuit breakers that exclude failing providers entirely, connection cooldowns that temporarily disable specific account keys, and model lockouts that isolate individual models while keeping connections active for other workloads.

OmniRoute is an open-source AI request routing system that implements sophisticated fault tolerance to handle transient failures and rate limits across multiple LLM providers. According to the diegosouzapw/OmniRoute source code, the platform uses a hierarchical protection system where circuit breakers, connection cooldowns, and model lockouts operate at different granularities to ensure high availability without over-penalizing healthy resources.

The Three Resilience Layers

OmniRoute implements defense in depth through three distinct mechanisms defined in src/shared/utils/circuitBreaker.ts and open-sse/services/accountFallback.ts.

Provider Circuit Breaker

The provider circuit breaker operates at the broadest scope, monitoring all accounts under a single provider (such as OpenAI or Gemini). Implemented in src/shared/utils/circuitBreaker.ts, this breaker trips when a provider repeatedly returns hard-failure status codes (408, 429, 500, 502, 503, 504). While the breaker state is OPEN, the provider is completely excluded from combo routing, regardless of individual connection health.

Connection Cooldown

The connection cooldown targets a single account or API key. After transient errors such as generic 429 rate limits or 500 server errors, the connection is marked unavailable for a calculated backoff period stored in rateLimitedUntil. The functions setConnectionRateLimitUntil and markAccountUnavailable in open-sse/services/accountFallback.ts handle this logic, ensuring all models on that specific connection are skipped until the cooldown expires.

Model Lockout

The model lockout provides the finest granularity, targeting a specific provider + connection + model combination. For providers enforcing per-model quotas (such as Gemini, NVIDIA, or Ollama-cloud), a failing model is locked out via lockModel while the underlying connection remains usable for other models. The isModelLocked and getModelLockoutInfo functions in open-sse/services/accountFallback.ts manage these restrictions.

How the Layers Interact in the Request Pipeline

Understanding the precedence and interaction rules is critical for debugging routing behavior in production environments.

Provider Circuit Breaker Dominates

If a provider's circuit breaker is OPEN, both connection cooldowns and model lockouts are ignored because the provider is already excluded from routing. This check occurs early in the request pipeline within src/sse/services/auth.ts and the combo candidate filter, ensuring no wasted requests are sent to unhealthy providers.

Connection Cooldown vs. Model Lockout

When a request fails, markAccountUnavailable determines whether to apply a connection-wide cooldown or a per-model lockout based on the error classification:

• Per-model errors (such as 404 for a single model or 429 classified as quota-exhausted) trigger lockModelIfPerModelQuota. The connection itself stays healthy, allowing other models on the same key to be used.
• Connection-wide errors (such as generic 429 rate-limits or 500 errors) trigger setConnectionRateLimitUntil, skipping all models on that connection until the cooldown expires.

Hybrid Error Classification

Some errors require additional inspection. For 429 responses containing a quota-exhausted body, the system calls shouldMarkAccountExhaustedFrom429. If the provider does not use per-model quotas (like standard OpenAI accounts), the error falls back to a connection cooldown; otherwise, it becomes a model lockout.

Combo Routing Skip Logic

Before attempting a target, the combo routing code in open-sse/services/combo.ts performs the following checks:

if (isModelLocked(provider, target.connectionId || "", rawModel)) {
    log.info("COMBO", `Skipping retry for ${modelStr} — model lockout active`);
    continue; // model is locked, connection stays usable
}
// later, after a failure:
if (fallbackResult.shouldCooldown) {
    // record a connection-wide cooldown
}

This logic ensures locked models are not retried while other models on the same connection remain candidates. The final skip decisions are implemented in open-sse/services/combo/targetResolution.ts.

Automatic Recovery

Both cooldowns and lockouts use lazy recovery. The next time the connection or model is inspected, the stored rateLimitedUntil or until timestamp is compared to Date.now(). If the time has passed, the entry clears automatically via clearAllModelLockouts or resetCooldownFailureCount.

Practical Code Examples

Checking System State

Inspect the current status of all three protection layers using the following utilities:

import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";
import { isModelLocked, getModelLockoutInfo } from "@omniroute/open-sse/services/accountFallback";

// Provider circuit-breaker status
const breaker = getCircuitBreaker("openai");
console.log(`OpenAI breaker state: ${breaker.state}`);

// Connection cooldown (rateLimitedUntil) – read directly from DB
const conn = await db.providers.getConnection("openai", connectionId);
console.log(`Connection cooldown until: ${conn.rateLimitedUntil}`);

// Model lockout
const locked = isModelLocked("openai", connectionId, "gpt-4o-mini");
console.log(`Model gpt-4o-mini locked? ${locked}`);

if (locked) {
  const info = getModelLockoutInfo("openai", connectionId, "gpt-4o-mini");
  console.log(`Lockout expires in ${info?.remainingMs} ms, reason: ${info?.reason}`);
}

Recording Per-Model Failures

When a per-model error occurs on a quota-enforced provider, trigger a model-specific lockout:

import { markAccountUnavailable } from "@omniroute/open-sse/services/auth";

// Simulate a 404 on a per-model-quota provider (e.g., Gemini)
await markAccountUnavailable(
  connectionId,
  404,
  "model not found",
  "gemini",
  "gemini-1.5-flash"
);
// Creates a model lockout (~30s by default) but leaves the
// connection usable for other Gemini models.

Triggering Connection-Wide Cooldowns

For errors affecting the entire connection, omit the model parameter to trigger a full cooldown:

await markAccountUnavailable(
  connectionId,
  429,
  "Too many requests",
  "openai"
  // No model argument → connection-wide cooldown
);

This sets rateLimitedUntil on the connection according to the provider profile (e.g., 60 seconds for OAuth, 30 seconds for API-key accounts).

Resetting Lockouts

Clear all active model lockouts programmatically without affecting connection cooldowns:

import { clearAllModelLockouts } from "@omniroute/open-sse/services/accountFallback";

// Clears every in-memory model lockout; connection cooldowns remain untouched.
clearAllModelLockouts();

Summary

• Provider circuit breakers act as the first line of defense, excluding entire providers when systemic failures occur (src/shared/utils/circuitBreaker.ts).
• Connection cooldowns isolate specific API keys after transient errors, managed by setConnectionRateLimitUntil in open-sse/services/accountFallback.ts.
• Model lockouts provide granular protection for per-model quotas via lockModel, allowing connections to serve other models while one model recovers.
• Interaction hierarchy places circuit breakers above cooldowns and lockouts; if a breaker is open, lower-level restrictions are irrelevant.
• Lazy recovery ensures system efficiency by clearing expired restrictions only when they are next accessed, avoiding background cleanup processes.

Frequently Asked Questions

What happens when a provider's circuit breaker is open but only one model is failing?

When a provider circuit breaker enters the OPEN state, the entire provider is excluded from routing regardless of individual model health. According to the implementation in src/sse/services/auth.ts, this check occurs before evaluating individual connections or models, meaning both connection cooldowns and model lockouts are effectively ignored until the breaker resets.

How does OmniRoute distinguish between a connection-wide cooldown and a per-model lockout?

The system distinguishes these based on error type and provider capabilities. In open-sse/services/accountFallback.ts, the markAccountUnavailable function checks if an error qualifies as per-model (such as 404 or quota-exhausted 429 responses) and whether the provider supports per-model quotas. If both conditions are true, it calls lockModelIfPerModelQuota; otherwise, it invokes setConnectionRateLimitUntil for the entire connection.

How long do connection cooldowns and model lockouts last?

Connection cooldowns use provider-specific durations defined in configuration profiles (typically 30 seconds for API-key accounts and 60 seconds for OAuth). Model lockouts default to approximately 30 seconds but vary based on the error classification. Both use timestamp-based expiration checked lazily against Date.now() when the resource is next accessed.

Can I manually reset a model lockout without restarting the service?

Yes. Import clearAllModelLockouts from @omniroute/open-sse/services/accountFallback to immediately clear all in-memory model lockouts. This function preserves connection cooldowns and circuit breaker states, allowing surgical recovery of specific models without affecting the broader system's failure tracking.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}