How OmniRoute Ensures Service Reliability: Circuit Breakers, Fallback Routing, and Persistent Health Monitoring

OmniRoute ensures service reliability through a multi-layered resiliency architecture featuring circuit breakers per provider, exponential backoff retries, automatic combo routing fallbacks, and persistent health monitoring.

OmniRoute is an open-source routing engine designed to maintain high availability across disparate AI providers and upstream services. To ensure the reliability of its services, the codebase implements a comprehensive suite of resiliency primitives that isolate failures, prevent cascading outages, and automatically reroute traffic without manual intervention. These mechanisms are implemented across the diegosouzapw/OmniRoute repository using TypeScript-based circuit breakers, intelligent retry policies, and SQLite-backed state persistence.

Circuit Breaker Pattern for Provider Isolation

OmniRoute implements a circuit breaker per provider to prevent repeated calls to failing upstream services. Located in src/shared/utils/circuitBreaker.ts, the CircuitBreaker class tracks consecutive failures and classifies them by kind—distinguishing between rate limits, quota exhaustion, and transient network errors.

Each breaker maintains three states:

• CLOSED: Normal operation; requests pass through.
• OPEN: The circuit is tripped after exceeding the configured failureThreshold; requests are short-circuited immediately to save latency.
• HALF_OPEN: After a configurable resetTimeout (exponential backoff), limited probe requests (halfOpenRequests) test the provider; successful probes close the circuit, while failures reopen it.

The implementation supports per-failure-kind thresholds and escalation warnings. Breaker states persist via the domainState database module, ensuring resilience survives application restarts.

import { getCircuitBreaker } from '@omniroute/shared/utils/circuitBreaker';

// Configure a breaker that trips after 3 failures and waits 30 s before probing.
const apiBreaker = getCircuitBreaker('my-api', {
  failureThreshold: 3,
  resetTimeout: 30_000,
  halfOpenRequests: 1,
  onStateChange: (name, oldState, newState) => {
    console.log(`Breaker ${name}: ${oldState} → ${newState}`);
  },
});

// Wrap an async call – the breaker will automatically retry or short-circuit.
async function callMyApi() {
  return apiBreaker.execute(async () => {
    const res = await fetch('https://example.com/endpoint');
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    return res.json();
  });
}

Automatic Retry with Exponential Backoff

Core executors in open-sse/executors/base.ts wrap every HTTP request in a robust retry loop via the execute() method. The retry policy respects Retry-After headers and distinguishes between idempotent-safe errors and fatal failures.

The tiered retry logic handles:

• Transient transport errors and early-EOF streams.
• Specific 5xx responses indicating temporary unavailability.
• Network timeouts with configurable exponential backoff.

This ensures that momentary blips do not trigger circuit breaker trips prematurely, while still surfacing persistent errors to the calling application.

Fallback Combo Routing for Continuous Availability

When a provider's circuit breaker opens, OmniRoute's combo routing engine in open-sse/services/combo.ts automatically selects an alternate target. The routing engine expands a combo configuration into an ordered list of ResolvedComboTarget objects based on the configured strategy—such as weighted-least-used or cost-optimized.

The system attempts each target in sequence until one succeeds, ensuring requests can be fulfilled even when primary providers degrade. This fallback mechanism works in concert with circuit breakers to create a self-healing traffic mesh.

import { resolveComboTargets, handleComboChat } from '@omniroute/open-sse/services/combo';

// Define a combo that prefers Provider A but falls back to Provider B.
const comboConfig = {
  strategy: 'weighted',
  targets: [
    { provider: 'provider-a', model: 'gpt-4o', weight: 0.7 },
    { provider: 'provider-b', model: 'claude-3', weight: 0.3 },
  ],
};

async function chat(requestBody) {
  const targets = resolveComboTargets(comboConfig);
  return handleComboChat(requestBody, targets);
}

Rate Limiting and Back-Pressure Management

Incoming requests are throttled by a sliding-window limiter implemented in src/lib/rateLimitManager.ts. This component enforces back-pressure to prevent overwhelming upstream providers.

When a provider returns a 429 status with a Retry-After header, the limiter records the cooldown period. Simultaneously, the circuit breaker classifies the failure kind, ensuring subsequent requests respect the provider's constraints and preventing hammering behavior that could trigger account bans.

Persistent State and Health Monitoring

Reliability requires observability. OmniRoute persists all breaker states via the domainState module in src/lib/db/domainState.ts, using SQLite for durable storage. A background sweeper runs every 5 minutes to evict idle, closed breakers, keeping the in-memory registry bounded to a maximum of 500 entries to prevent memory leaks.

Dedicated health-check loops in src/lib/healthCheck.ts periodically probe providers, while the token refresh service in src/lib/tokenRefreshService.ts uses its own circuit breaker to abort refresh attempts after repeated 401/403 failures, preventing endless credential churn.

Operators can inspect system health via the MCP diagnostics tool:

import { getAllCircuitBreakerStatuses } from '@omniroute/shared/utils/circuitBreaker';

console.log(getAllCircuitBreakerStatuses());
// → [{ name: 'my-api', state: 'CLOSED', failureCount: 0, ... }, …]

Every state transition is recorded in transitionHistory, exposing the current health of each provider, back-off timers, and degradation reasons.

Summary

• Circuit breakers isolate provider failures per upstream service, with automatic state transitions between CLOSED, OPEN, and HALF_OPEN.
• Exponential backoff retry logic in the base executor handles transient errors while respecting Retry-After headers.
• Combo routing provides automatic fallback to healthy providers when primary targets are unavailable.
• Rate limiting coordinates with circuit breakers to enforce back-pressure and prevent quota exhaustion.
• Persistent SQLite storage and a bounded in-memory registry ensure reliability state survives restarts without unbounded memory growth.
• Health monitoring and circuit-breaker-protected token refresh services prevent cascading authentication failures.

Frequently Asked Questions

What happens when a circuit breaker enters the HALF_OPEN state?

When a circuit breaker transitions from OPEN to HALF_OPEN after the configured resetTimeout, it allows a limited number of probe requests (controlled by halfOpenRequests) to test the provider's health. If a probe succeeds, the breaker closes the circuit and resumes normal traffic; if it fails, the breaker returns to OPEN and resets the backoff timer.

How does OmniRoute handle rate limit responses from upstream providers?

The RateLimitManager in src/lib/rateLimitManager.ts implements a sliding-window limiter that throttles incoming requests. When a provider returns a 429 status with a Retry-After header, the system records the cooldown period and the circuit breaker classifies the failure kind, ensuring subsequent requests respect the provider's limits and avoiding hammering behavior.

What is the maximum number of circuit breakers that can be tracked simultaneously?

The in-memory registry is bounded to a maximum of 500 entries to prevent memory leaks. A background sweeper runs every 5 minutes to evict idle, closed breakers from the registry, while persistent state is maintained in SQLite via the domainState module in src/lib/db/domainState.ts.

How does combo routing select an alternative provider when the primary fails?

The combo routing engine in open-sse/services/combo.ts expands a configuration into an ordered list of ResolvedComboTarget objects based on the configured strategy (e.g., weighted-least-used). When the primary provider's circuit breaker is open, the system automatically attempts the next target in the list until one succeeds or the list is exhausted.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}