Understanding OmniRoute's 3-Layer Resilience Mechanism: Provider, Connection & Model-Level Protection

OmniRoute implements a three-layer resilience system—provider-level circuit breakers, connection-level cooldowns, and model-level lockouts—to isolate failures and maintain request throughput without cascading outages.

OmniRoute's 3-layer resilience mechanism is designed to handle failures at multiple granularities, from entire AI providers down to individual models on specific API keys. This architecture ensures that a single point of failure—whether it's a provider outage, a rate-limited credential, or a quota-exhausted model—does not degrade overall routing performance. Below, we examine how each layer operates, where the code lives, and how they interact during request processing.

---

Provider-Level Circuit Breaker

The provider-level circuit breaker sits at the top of the resilience stack, monitoring all traffic to a given provider (e.g., openai, anthropic). When failures exceed configurable thresholds, the breaker opens and halts all requests to that provider.

State Machine Implementation

In src/shared/utils/circuitBreaker.ts (lines 5–17), the breaker implements a classic CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED state machine. Key capabilities include:

• Failure-type awareness: The PROVIDER_FAILURE_ERROR_CODES array (408, 429, 500, 502, 503, 504) defines which HTTP status codes count toward the failure threshold.
• Adaptive back-off: After each OPEN → HALF_OPEN → OPEN cycle, the _effectiveResetTimeout escalates up to a configurable multiplier (lines 55–64).
• Per-failure-kind thresholds: The kindThresholds map (lines 78–84) allows distinct thresholds for rate_limit, quota_exhausted, and other failure types, including immediate-open behavior for critical errors.
• State persistence: Breaker state survives restarts via saveCircuitBreakerState and loadCircuitBreakerState (lines 19–24).

The public API exposes recordProviderFailure and recordProviderSuccess (lines 87–95 and 52–57), which automatically deduplicate rapid-fire failures per connection and aggregate provider-wide network errors.

---

Connection-Level Cooldown

The connection-level cooldown provides finer-grained protection than the provider breaker. It targets individual credentials, API keys, or OAuth accounts while allowing other connections for the same provider to continue serving requests.

How It Works

Each connection record stores a rateLimitedUntil timestamp in the database (defined in the AccountState type). When a request fails for a specific key, setConnectionRateLimitUntil advances this timestamp:

• 5 seconds for OAuth-based connections
• 3 seconds for API-key-based connections

The router skips connections where new Date(rateLimitedUntil).getTime() > Date.now() (see open-sse/services/accountFallback.ts, lines 10, 20–27).

The cooldown mechanism is lazy: no background process re-enables connections. When the timestamp passes, the connection becomes eligible automatically. Repeated failures trigger exponential back-off via calculateBackoffCooldown, capped at maxCooldownMs from the PROVIDER_PROFILE configuration.

---

Model-Level Lockout

The model-level lockout is the most granular layer. It prevents a single unavailable or quota-limited model from disabling an entire connection containing multiple models.

Lockout Lifecycle

In open-sse/services/accountFallback.ts (lines 53–61), the lockModel function creates entries tracking:

• reason: The failure type (e.g., quota_exhausted, model_not_found)
• until: Expiration timestamp
• failureCount: Accumulated failures for backoff calculation

The cleanup timer (ensureCleanupTimer, lines 17–33) automatically removes expired locks. Public functions isModelLocked, getModelLockoutInfo, and clearModelLock expose lock state to dashboards and monitoring tools.

This layer is especially critical for providers with per-model quotas (Gemini, Codex, Antigravity), where one model's exhaustion should not affect others on the same credential.

---

How the Three Layers Interact

During request processing, OmniRoute evaluates resilience layers in strict order:

1. Combo routing generates candidate targets.
2. For each target, accountFallback checks:
   • Provider breaker: isProviderInCooldown (line 57) aborts if the breaker is OPEN.
   • Connection cooldown: Skips if rateLimitedUntil is in the future (line 10).
   • Model lockout: isModelLocked (line 36) blocks specific model-connection pairs.
3. If the request fails, the appropriate layer records it:
   • recordProviderFailure → circuit breaker
   • setConnectionRateLimitUntil → connection cooldown
   • recordModelLockoutFailure (lines 19–28) → model lockout
4. The router retries the next candidate, automatically respecting new cooldowns and locks.

Refer to docs/diagrams/resilience-3layers.svg for a visual representation of this flow.

---

Practical Code Examples

Check if a provider is in cooldown

import { isProviderInCooldown } from '@/lib/usage/accountFallback';

if (isProviderInCooldown('openai')) {
  // Skip all OpenAI targets for this request
}

Record a connection-level rate limit

import { setConnectionRateLimitUntil } from '@/lib/db/providers';
import { getConnectionById } from '@/lib/db/connections';

async function onRateLimited(connectionId: string) {
  const conn = await getConnectionById(connectionId);
  const cooldownMs = conn.category === 'oauth' ? 5000 : 3000;
  setConnectionRateLimitUntil(connectionId, Date.now() + cooldownMs);
}

Lock a model after quota exhaustion

import { lockModel } from '@/lib/usage/accountFallback';

function handleQuotaExhausted(provider: string, connectionId: string, model: string) {
  const ONE_HOUR = 60 * 60 * 1000;
  lockModel(provider, connectionId, model, 'quota_exhausted', ONE_HOUR);
}

Query active model lockouts for dashboards

import { getAllModelLockouts } from '@/lib/usage/accountFallback';

const lockouts = getAllModelLockouts();
lockouts.forEach(l => {
  console.log(`${l.provider}:${l.connectionId}:${l.model} locked for ${l.reason}, ${l.remainingMs}ms left`);
});

---

Key Source Files

File	Purpose
src/shared/utils/circuitBreaker.ts	Provider-level circuit breaker: state machine, persistence, per-failure-kind thresholds.
open-sse/services/accountFallback.ts	Connection cooldown, model lockouts, provider profile resolution, router integration.
docs/diagrams/resilience-3layers.svg	Visual diagram of layer interactions.
src/lib/resilience/settings.ts	Default resilience configuration: cooldown durations, thresholds, back-off steps.
src/lib/resilience/modelLockoutSettings.ts	Per-model lockout settings and maximum cooldown limits.

---

Summary

• Provider-level circuit breakers isolate entire providers during systemic failures, with adaptive back-off and state persistence across restarts.
• Connection-level cooldowns temporarily disable individual credentials while preserving provider access through alternate keys.
• Model-level lockouts prevent single-model issues from poisoning multi-model connections.

These layers operate sequentially during routing decisions and record failures independently, enabling automatic recovery without manual intervention.

---

Frequently Asked Questions

What triggers a provider-level circuit breaker to open?

The circuit breaker opens when failures exceeding the configured threshold occur within a time window. In src/shared/utils/circuitBreaker.ts, failures are counted for HTTP status codes in PROVIDER_FAILURE_ERROR_CODES (408, 429, 500, 502, 503, 504). Per-failure-kind thresholds in kindThresholds (lines 78–84) allow quota_exhausted or rate_limit errors to trigger faster opens than transient network errors.

How does OmniRoute handle multiple API keys for the same provider?

Each key is a separate connection with its own rateLimitedUntil timestamp. When one key hits a rate limit, setConnectionRateLimitUntil disables only that connection (typically 3–5 seconds), while other connections for the same provider continue serving requests. This is implemented in open-sse/services/accountFallback.ts (lines 20–27).

Can model lockouts persist longer than connection cooldowns?

Yes. Model lockouts default to longer durations—often 1 hour for quota_exhausted—while connection cooldowns typically cap at seconds or minutes. The lockModel function (lines 53–61) accepts arbitrary until timestamps, and modelLockoutSettings.ts configures maximum limits per provider.

What happens to in-flight requests when a breaker opens?

In-flight requests complete normally; the breaker only affects new routing decisions. The isProviderInCooldown check (line 57) filters targets before request dispatch, so open breakers cause immediate failover to alternate providers without aborting active connections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}