How Circuit Breakers Are Implemented in OmniRoute: A Deep Dive into the Resilience Architecture

OmniRoute implements circuit breakers at the provider level through a four-state finite state machine in src/shared/utils/circuitBreaker.ts, with thresholds configurable per provider type to isolate failing upstream services before they cascade through the routing layer.

The circuit breaker pattern in OmniRoute is a critical resilience mechanism that prevents the AI gateway from repeatedly calling unhealthy LLM providers. Unlike simple timeout-based retries, this implementation uses a sophisticated state machine with an early-warning DEGRADED state to give operators visibility before full isolation occurs. The pattern is integrated directly into the server-sent event (SSE) request handlers and exposed through a health monitoring API.

How the OmniRoute Circuit Breaker State Machine Works

The core implementation lives in src/shared/utils/circuitBreaker.ts and defines four distinct states that control traffic flow to each provider:

State	Behavior	Transition Trigger
CLOSED	Normal operation; all requests allowed	Accumulated failures reach degradation or open threshold
DEGRADED	Traffic continues; UI surfaces warning	Failure count exceeds circuitBreakerDegrade but below open threshold
OPEN	Provider blocked; requests rejected or skipped	Consecutive failures hit circuitBreakerThreshold
HALF_OPEN	Single probe request allowed to test recovery	Reset timeout expires automatically

State transitions happen lazily. When canExecute() is called, the breaker checks Date.now() against the last failure timestamp plus the reset timeout. If an OPEN breaker has exceeded its cooldown, it automatically transitions to HALF_OPEN without requiring external coordination.

Configuration Parameters and Provider Profiles

Breaker thresholds are defined in src/shared/constants/providers.ts and exposed through environment variables with the prefix OMNIROUTE_PROVIDER_BREAKER_*:

Parameter	Environment Variable	Purpose
circuitBreakerThreshold	OMNIROUTE_PROVIDER_BREAKER_THRESHOLD	Consecutive failures before opening the breaker
circuitBreakerReset	OMNIROUTE_PROVIDER_BREAKER_RESET	Seconds until OPEN → HALF_OPEN transition
circuitBreakerDegrade	OMNIROUTE_PROVIDER_BREAKER_DEGRADE	Failure count that triggers DEGRADED state

Default profiles vary by provider authentication type. OAuth providers like OpenAI use stricter defaults (8 failures, 60s reset), while local or API-key providers may tolerate more failures before degradation.

// src/shared/constants/providers.ts (conceptual structure)
export interface ProviderCircuitConfig {
  circuitBreakerThreshold: number;
  circuitBreakerReset: number;
  circuitBreakerDegrade: number;
}

export const DEFAULT_OAUTH_PROVIDER: ProviderCircuitConfig = {
  circuitBreakerThreshold: 8,
  circuitBreakerReset: 60,
  circuitBreakerDegrade: 4,
};

Integrating Circuit Breakers in Request Handlers

The breaker is consulted on every routed request through the getCircuitBreaker() factory. In src/sse/handlers/chatHelpers.ts, the handler obtains a breaker instance scoped to the target provider:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { ProviderName } from '@/shared/constants/providers';

// Executed per incoming chat request
const breaker = getCircuitBreaker(ProviderName.OPENAI);

if (!breaker.canExecute()) {
  // Provider is OPEN or in cooldown - routing logic skips to fallback
  // Returns standardized error: 'provider-circuit-open'
  return { error: 'provider-circuit-open', status: 503 };
}

// Proceed with upstream request

After the upstream responds, the handler updates breaker state:

// Successful response path
breaker.recordSuccess();  // Resets failure accumulator to zero

// Failure path - only server errors count toward breaker
const SERVER_ERROR_CODES = new Set([408, 500, 502, 503, 504]);

if (SERVER_ERROR_CODES.has(response.status)) {
  breaker.recordFailure();  // May trigger DEGRADED or OPEN
}

What Errors Actually Trip the Circuit Breaker

Not all failures are created equal in OmniRoute's implementation. The breaker only reacts to server-side transport errors:

• Trips the breaker: 408 Request Timeout, 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout

• Ignored by breaker: 401 Unauthorized, 403 Forbidden, 429 Too Many Requests

Authentication failures and rate limits are handled through separate mechanisms—connection cooldowns for 429s and model lockouts for quota exhaustion. This separation prevents transient user configuration errors from unnecessarily isolating entire providers.

Monitoring Circuit Breaker State at Runtime

The breaker exposes internal state through src/app/api/monitoring/health/route.ts, enabling real-time observability:

// Aggregate breaker status across all configured providers
const healthStatus = {
  providers: Object.values(ProviderName).map(name => {
    const breaker = getCircuitBreaker(name);
    return {
      name,
      state: breaker.getState(),        // 'CLOSED' | 'DEGRADED' | 'OPEN' | 'HALF_OPEN'
      failures: breaker.getFailureCount(),
      lastFailure: breaker.getLastFailureTime(),
      nextRetry: breaker.getNextRetryTime(),
    };
  })
};

This endpoint drives operational dashboards and can trigger pager duty alerts when critical providers enter DEGRADED or OPEN states.

Summary

• Core Location: The CircuitBreaker class in src/shared/utils/circuitBreaker.ts implements a four-state machine with lazy timeout-driven transitions.

• Configuration: Three environment-controlled parameters (circuitBreakerThreshold, circuitBreakerReset, circuitBreakerDegrade) define provider-specific resilience profiles in src/shared/constants/providers.ts.

• Integration Point: Request handlers in src/sse/handlers/chatHelpers.ts obtain breakers via getCircuitBreaker() and consult canExecute() before upstream calls.

• Selective Failure Handling: Only HTTP 408, 500, 502, 503, 504 errors increment the breaker; auth and rate-limit errors use separate mitigation paths.

• Observability: Runtime state is queryable through the health monitoring API at src/app/api/monitoring/health/route.ts.

Frequently Asked Questions

How does OmniRoute's circuit breaker differ from a simple retry loop with exponential backoff?

OmniRoute's breaker is stateful and provider-scoped, not request-scoped. While exponential backoff retries a single request multiple times, the circuit breaker blocks all traffic to a provider after threshold failures, preventing wasted resources and cascading latency. The DEGRADED state provides early warning without full isolation—something pure retry logic cannot offer.

Can I disable the circuit breaker for specific providers?

Yes. Set circuitBreakerThreshold to Infinity or a sufficiently high value in your provider configuration within src/shared/constants/providers.ts. The breaker will still track failures but never transition to OPEN. Note that this removes protection against upstream outages.

What happens to requests when a provider's circuit breaker is OPEN?

Requests receive a structured error response with code provider-circuit-open and HTTP status 503. The routing layer may then apply combo routing logic to failover to alternative providers, or return the error to the client depending on configuration.

How is the HALF_OPEN state tested without affecting production traffic?

When timeout expires, canExecute() permits exactly one request through. Success via recordSuccess() closes the breaker immediately. Failure via recordFailure() re-opens it for another full reset period. This probe mechanism requires no separate "canary" infrastructure—it piggybacks on real user traffic with automatic rollback.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}