How OmniRoute Manages Model Lifecycle Policies: A Three-Layer Resilience System

OmniRoute v3.8.51 implements a three-layer resilience system—provider circuit breaker, connection cooldown, and model lockout—to dynamically govern model lifecycle policies from request validation through provider execution.

OmniRoute's model lifecycle policies prevent cascading failures by isolating problems at different granularities. Rather than treating all errors equally, the system distinguishes between provider-wide outages, individual connection issues, and model-specific failures. This architecture ensures that a single bad API key or exhausted quota won't cripple the entire routing pipeline.

Three-Layer Policy Architecture

OmniRoute organizes its lifecycle policies into three distinct layers, each targeting a different scope of failure.

Layer	Scope	Purpose
Provider Circuit Breaker	Entire provider (e.g., openai, anthropic)	Halts traffic to consistently failing upstreams
Connection Cooldown	Individual connection/account/key	Temporarily skips bad credentials while preserving other connections
Model Lockout	Provider + connection + specific model	Isolates per-model failures like quota limits or missing deployments

The routing layer in open-sse/services/combo.ts consults all three layers before selecting a target, automatically filtering out any provider, connection, or model that violates current policy state.

Provider Circuit Breaker: System-Wide Protection

The provider circuit breaker stops sending traffic to an entire provider when upstream failures exceed configured thresholds. This protects OmniRoute from wasting resources on endpoints that are fundamentally unreachable.

Implementation Details

The circuit breaker lives in src/shared/utils/circuitBreaker.ts and implements a four-state machine:

• CLOSED — Normal operation, requests allowed
• DEGRADED — Partial failure, limited traffic permitted
• OPEN — Circuit tripped, all requests blocked
• HALF_OPEN — Testing recovery with a single probe request

Configuration resides in PROVIDER_PROFILES within open-sse/config/constants.ts. The src/sse/handlers/chatHelpers.ts module applies these profiles during request validation.

// Checking if a provider is allowed to execute
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { Provider } from '@/shared/constants/providers';

async function canExecute(provider: Provider): Promise<boolean> {
  const breaker = getCircuitBreaker(provider);
  // State refreshes lazily from timestamps on each read
  return breaker.canExecute(); // true if CLOSED/DEGRADED, false if OPEN
}

The breaker uses lazy state evaluation—timestamps like rateLimitedUntil are checked on every canExecute() call rather than relying on background timers. This eliminates coordination overhead while enabling automatic recovery.

Connection Cooldown: Granular Credential Isolation

When a specific API key or account hits rate limits, the connection cooldown layer marks that individual connection unavailable without affecting other connections to the same provider.

Key Implementation Files

• src/sse/services/auth.ts::markAccountUnavailable() — triggers cooldown after failure detection
• open-sse/services/accountFallback.ts::checkFallbackError() — validates cooldown status before routing
• src/lib/resilience/settings.ts — stores cooldown durations and retry policies

// Marking an account unavailable after a 429 response
import { markAccountUnavailable } from '@/sse/services/auth';

async function handleRateLimited(err: any, accountId: string) {
  if (err.status === 429) {
    await markAccountUnavailable(accountId, {
      retryAfterMs: err.retryAfter ?? 3000, // 3-second fallback
    });
  }
}

Cooldown state persists as timestamps, allowing automatic expiration without scheduled jobs. A connection becomes eligible for retry once its retryAfterMs period elapses.

Model Lockout: Fine-Grained Failure Isolation

The most granular layer, model lockout, handles failures specific to individual models within a connection—such as per-model quota exhaustion or deployments that no longer exist.

Lockout Logic

Implemented in open-sse/services/accountFallback.ts, model lockout prevents the routing layer from selecting compromised models while preserving healthy ones on the same connection. The combo routing logic consults this state before finalizing target selection.

// Disabling a specific model after quota exhaustion
import { disableModelLockout } from '@/open-sse/services/accountFallback';

async function handleModelQuota(err: any, modelId: string, connectionId: string) {
  if (err.status === 429 && err.model === modelId) {
    await disableModelLockout(connectionId, modelId);
  }
}

Model lockout records per-model error flags that persist independently of connection cooldown state. This means a 429 on gpt-4-turbo won't disable gpt-4o on the same credential.

Policy Engine Integration

Higher-level services interact with lifecycle policies through src/domain/policyEngine.ts, which abstracts the three layers into a unified query interface. This abstraction allows routing components to check policy compliance without directly managing state machines or timestamp arithmetic.

The policy engine's lazy evaluation design ensures zero background overhead—no timers, no polling loops, no goroutines. State transitions happen inline with request processing, and expiration is computed on demand.

Thresholds and Configuration

Default thresholds live in src/lib/resilience/settings.ts and can be overridden via environment variables:

Failure Type	Typical Threshold	Action
Provider HTTP 5xx	5 errors in 60 seconds	Circuit breaker → OPEN
Connection 429	3 consecutive rate limits	Cooldown for retryAfter duration
Model-specific 429	1 error with model attribution	Model lockout for 5 minutes

These defaults balance rapid failure detection against unnecessary isolation of transient issues.

Code Examples: Complete Policy Flow

// Full request pipeline with policy checks
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { checkFallbackError } from '@/open-sse/services/accountFallback';
import { selectComboTarget } from '@/open-sse/services/combo';

async function routeRequest(request: ChatRequest) {
  // Layer 1: Provider-level check
  const providerBreaker = getCircuitBreaker(request.provider);
  if (!await providerBreaker.canExecute()) {
    throw new ProviderUnavailableError(request.provider);
  }

  // Layer 2 & 3: Connection and model checks happen inside combo routing
  const target = await selectComboTarget(request, {
    filterCooldown: true,
    filterLockedModels: true,
  });

  if (!target) {
    throw new NoAvailableModelsError();
  }

  return target;
}

The selectComboTarget function encapsulates the interaction between connection cooldown and model lockout layers, returning only targets that satisfy all active policies.

Summary

OmniRoute's model lifecycle policy system provides:

• Three isolation layers protecting against provider, connection, and model-level failures
• Lazy state evaluation eliminating background timers and enabling automatic recovery
• Configurable thresholds via src/lib/resilience/settings.ts and environment variables
• Unified policy engine abstraction in src/domain/policyEngine.ts for clean integration

These mechanisms ensure that upstream outages, rate limits, and model-specific issues are contained without manual intervention or global service degradation.

Frequently Asked Questions

What triggers a provider circuit breaker to open?

The provider circuit breaker transitions to OPEN when HTTP 5xx errors exceed the threshold defined in PROVIDER_PROFILES—typically 5 errors within 60 seconds. Once open, the breaker blocks all traffic to that provider until the HALF_OPEN probe succeeds or the timeout expires.

How does OmniRoute recover from a connection cooldown automatically?

Connection cooldowns store rateLimitedUntil timestamps that are checked lazily on each routing decision. No background process monitors expiration; instead, checkFallbackError() in open-sse/services/accountFallback.ts validates whether the cooldown period has elapsed before filtering out a connection.

Can model lockout affect other connections using the same provider?

No. Model lockout is scoped to a specific connectionId + modelId pair as implemented in open-sse/services/accountFallback.ts. A gpt-4 lockout on one API key does not impact gpt-4 availability on other credentials or providers.

Where are default resilience thresholds configured?

Default thresholds for all three policy layers reside in src/lib/resilience/settings.ts, with provider-specific profiles in open-sse/config/constants.ts. These values can be overridden through environment variables without code changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}