Connection Cooldown vs Model Lockout in OmniRoute: Key Differences Explained

Connection Cooldown temporarily disables an entire provider connection (account + API key) after transient HTTP errors, while Model Lockout isolates only a specific model on that connection, allowing other models to continue serving requests.

OmniRoute implements a three-tier resilience strategy to protect AI provider integrations from cascading failures. While the Provider Circuit Breaker acts as a coarse-grained safety net for entire providers, Connection Cooldown and Model Lockout operate at finer granularity to handle connection-wide and model-specific issues respectively. Understanding the distinction between these mechanisms is essential for debugging routing decisions and optimizing retry behavior in production deployments.

What Is Connection Cooldown?

Connection Cooldown guards a single connection—defined as a specific account and API key pair—to a provider. It triggers when the connection experiences retryable failures such as HTTP 408 (Request Timeout), 429 (Too Many Requests), or any 5xx server error.

When activated, OmniRoute sets a rateLimitedUntil timestamp on the connection. The routing layer in src/sse/services/auth.ts checks this timestamp via parseFutureDateMs(connection.rateLimitedUntil) and excludes the connection from selection while the cooldown is active. Other connections to the same provider remain fully operational.

The cooldown duration is calculated using exponential backoff (baseCooldownMs * 2**failureCount) and can optionally honor upstream Retry-After headers when useUpstreamRetryHints is enabled in the resilience profile.

What Is Model Lockout?

Model Lockout operates at a more granular level, guarding an individual model (or model family) on a specific connection. It triggers on per-model quota exhaustion (e.g., 429 errors with reason rate_limited), missing model errors (404), or provider-specific model-level limits.

The mechanism uses a composite key format "provider:connectionId:model" stored in a lockout map within open-sse/services/accountFallback.ts. When lockModel() is invoked, it stores an expiry timestamp (newUntil = Date.now() + cooldownMs) with optional metadata. The routing layer checks isModelLocked(provider, connectionId, requestedModel) before selecting a connection, and locked models are tracked separately via modelLockedCount with detailed logging of remaining lockout time.

Model lockouts are cleaned up automatically every 15 seconds via a lazy timer to prevent memory leaks.

Key Differences Between Connection Cooldown and Model Lockout

Aspect	Connection Cooldown	Model Lockout
Scope	Entire connection (account + key)	Single model on a specific connection
Trigger	HTTP 408/429/5xx, connection-level errors	Per-model quota limits, 404 missing model
Impact	All models on the connection are unavailable	Only the specific model is unavailable; other models work
Storage	rateLimitedUntil timestamp on connection object	Entry in "provider:connectionId:model" map
Backoff	Exponential with configurable steps	Fixed duration based on error context

How Connection Cooldown Works Under the Hood

Configuration and Detection

The resilience profile builder in open-sse/services/accountFallback.ts (lines 70-85) reads connection-level fields from settings including baseCooldownMs, useUpstreamRetryHints, and maxBackoffSteps. When a request fails with a retryable status, the connection's rateLimitedUntil field is updated to now + cooldownMs.

Cooldown Calculation

The cooldown calculation (lines 381-385 in open-sse/services/accountFallback.ts) supports exponential backoff and upstream hint processing:

// Exponential backoff: baseCooldownMs * 2^failureCount
const cooldownMs = baseCooldownMs * Math.pow(2, failureCount);

// Respect upstream Retry-After if enabled
if (useUpstreamRetryHints && retryAfterHeader) {
  return Math.max(cooldownMs, parseRetryAfter(retryAfterHeader));
}

Code Example

// Example: Connection enters cooldown after 429 error
await fetchProvider(request); // fails with HTTP 429

// OmniRoute automatically sets:
connection.rateLimitedUntil = Date.now() + cooldownMs;

// Subsequent routing checks skip this connection until timestamp expires

How Model Lockout Works Under the Hood

The Lockout Map Structure

Model lockouts are managed in open-sse/services/accountFallback.ts (lines 18-22) using a Map with composite keys:

// Key format: "provider:connectionId:model"
const lockoutKey = `${provider}:${connectionId}:${model}`;

// Stored value includes expiry and metadata
lockoutMap.set(lockoutKey, {
  until: Date.now() + cooldownMs,
  reason: 'rate_limited'
});

Routing Integration

The src/sse/services/auth.ts file (lines 12-16 and 1278-1297) implements the lockout check:

if (isModelLocked(provider, connectionId, requestedModel)) {
  modelLockedCount++;
  continue; // Skip this connection for this specific model
}

When all connections are blocked by model cooldowns for a requested model, the system reports allBlockedByModelCooldown (lines 19-26), setting cooldownScope to "model" in API responses.

Code Example

// Example: Locking a specific model on a connection
lockModel(
  "openai",
  "conn-1234",
  "gpt-4o-mini",
  "rate_limited",      // reason from RateLimitReason enum
  30_000               // 30-second lockout duration
);

// Later requests for "gpt-4o-mini" on conn-1234 will fail isModelLocked() check
// But requests for "gpt-4o" on the same connection proceed normally

When to Use Each Mechanism

Use Connection Cooldown when encountering generic transient errors that affect the whole account, such as rate-limit headers applied to the API key or temporary provider outages. This protects the routing engine from repeatedly hammering a flaky connection without penalizing specific models.

Use Model Lockout when errors are isolated to a single model, such as quota exhaustion for gpt-4o-mini while gpt-4o remains available, or when a model is removed from the provider's catalog. This allows the connection to continue serving other models while isolating the problematic one.

Summary

• Connection Cooldown operates at the connection level (account + API key), triggered by HTTP 408/429/5xx errors, and temporarily disables the entire connection using exponential backoff and optional Retry-After support.
• Model Lockout operates at the model level on a specific connection, triggered by per-model quota errors or 404s, using a composite key map to exclude only the affected model while preserving connection availability for other models.
• Both mechanisms are orthogonal to the Provider Circuit Breaker and cooperate to provide fine-grained resilience, with cooldownScope fields indicating whether a routing failure stems from connection or model-level blocks.

Frequently Asked Questions

Can a connection be in cooldown while a model is locked on another connection?

Yes. These mechanisms are independent. Connection A can be in cooldown (affecting all its models) while Connection B has Model X locked (but still serves Model Y). Additionally, Model X could be locked on Connection B even while Connection A (which is not in cooldown) continues to serve Model X normally.

How long do lockouts typically last?

Connection Cooldown durations are configured via baseCooldownMs in src/lib/resilience/settings.ts and grow exponentially with repeated failures (up to maxBackoffSteps). Model Lockout durations are typically shorter and context-specific—often 30-60 seconds for rate limits—though the exact value depends on the error response and hardcoded defaults in the resilience profile.

Does Model Lockout affect all connections or just one?

Model Lockout affects only the specific connection where the error occurred. The lockout key includes connectionId, meaning gpt-4o-mini might be locked on Connection A but fully available on Connection B. This isolation prevents a single model's quota exhaustion from degrading service for that model across your entire OmniRoute deployment.

Where are these resilience settings configured?

Default settings for both mechanisms reside in src/lib/resilience/settings.ts, including baseCooldownMs, useUpstreamRetryHints, and cleanup intervals. Runtime behavior is further controlled by the resilience profile builder in open-sse/services/accountFallback.ts. For human-readable documentation, refer to docs/architecture/RESILIENCE_GUIDE.md sections "2. Connection Cooldown" and "3. Model Lockout".

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}