# OmniRoute Circuit Breaker Lazy Recovery: Self-Healing Without Background Timers

> Discover OmniRoute circuit breaker lazy recovery: avoid background timers, refresh state on demand, and ensure precise cooldown expiration for self-healing.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-24

---

**OmniRoute implements lazy recovery by refreshing circuit breaker state only when status methods are called, eliminating background timers while ensuring providers transition from OPEN to HALF_OPEN precisely when cooldown periods expire.**

OmniRoute protects its LLM routing pipeline from cascading failures using a sophisticated circuit breaker implementation. Unlike traditional approaches that rely on background scheduling threads, the `CircuitBreaker` class in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) employs a **lazy recovery** mechanism that checks state freshness exclusively on demand. This design reduces resource overhead while maintaining strict consistency between the breaker's internal state and observed behavior.

## The Five-State Circuit Breaker Lifecycle

The implementation manages provider health through a strict state machine that includes an innovative intermediate status. According to the source code at lines 12–18, the breaker cycles through `CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED`.

### Degradation as an Early Warning

When the failure count exceeds a configurable **degradation threshold** (approximately 60% of the normal failure threshold), the breaker enters the `DEGRADED` state. This provides an early warning mechanism before the circuit fully opens, allowing the routing engine to deprioritize struggling providers without completely removing them from the pool.

### Kind-Specific Failure Classification

Errors are classified by *kind*—including rate-limit, quota-exhausted, and transient failures. As implemented in lines 77–85, per-kind thresholds can trigger an immediate transition to `OPEN`, or apply custom cooldown periods based on the error semantics. This granularity prevents temporary rate limits from being treated identically to persistent authentication failures.

## How Lazy Recovery Works

The lazy recovery mechanism eliminates the need for `setInterval` or background worker threads that periodically check timeout conditions. Instead, state transitions rely entirely on **read-time evaluation**.

### On-Demand State Refreshing

The private method `_refreshOpenState()` (lines 48–52) performs all timeout checking. This method is invoked at the start of every public status query: `canExecute()`, `getStatus()`, and `getRetryAfterMs()`. When any component consults the breaker, the elapsed time since the last failure is compared against `_effectiveCooldown()`. If the timeout has not elapsed, the breaker remains `OPEN`. If the timeout has passed, the state transitions to `HALF_OPEN` immediately.

### Transition to HALF_OPEN

When `_shouldAttemptReset()` confirms that elapsed time exceeds the effective cooldown, `_refreshOpenState()` calls `_transition(STATE.HALF_OPEN, "timeout-elapsed")`. The breaker then permits a limited number of probe requests controlled by the `halfOpenRequests` configuration parameter. A successful probe triggers `_onSuccess()`, which closes the circuit; a failure invokes `_onFailure()` and reopens the circuit, potentially with an increased back-off calculated by `_effectiveResetTimeout()`.

### SQLite Persistence for Process Survival

Breaker state—including last failure time, cumulative count, and error kind—is stored in the domain-state SQLite table via [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts). This persistence layer ensures that circuit status survives application restarts, preventing a restarted process from prematurely retrying a provider that triggered a cooldown just before termination.

## Pipeline Integration and Usage Patterns

Components interact with the breaker through the `getCircuitBreaker()` factory, checking availability before executing upstream calls.

```typescript
// Typical usage inside the request pipeline
const breaker = getCircuitBreaker(providerName);
if (!breaker.canExecute()) {
  // Provider is OPEN – skip it or fall back to another target
  throw new CircuitBreakerOpenError(
    `Provider ${providerName} circuit open`,
    providerName,
    breaker.getRetryAfterMs(),
  );
}

// Execute the LLM request
const response = await llmProvider.call(prompt);

// Record outcome
breaker.recordSuccess(); // internally calls _onSuccess()

```

The first line of `canExecute()` invokes `this._refreshOpenState()`, guaranteeing that any caller sees the most current state without additional scheduling overhead.

## Observability via Health Endpoints

The lazy recovery mechanism enables real-time visibility through standard monitoring interfaces. The health route at [`src/app/api/v1/monitoring/health/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/monitoring/health/route.ts) exposes breaker status via the `getStatus()` method, which itself triggers state refresh.

```typescript
// Lazy recovery in action – called via health endpoint
app.get("/api/v1/monitoring/health", async (req, res) => {
  const providers = Object.keys(providerRegistry);
  const breakerStatuses = providers.map((p) => getCircuitBreaker(p).getStatus());
  res.json({ providers: breakerStatuses });
});

```

Because status checks trigger `_refreshOpenState()`, the health endpoint always reports accurate recovery timing without polling or push-based updates.

## Summary

- **Lazy recovery** eliminates background timers by checking cooldown expiration only when `canExecute()`, `getStatus()`, or `getRetryAfterMs()` are called.
- The **five-state lifecycle** includes a `DEGRADED` warning state triggered at approximately 60% of the failure threshold, preventing premature circuit opening.
- **Kind-specific handling** allows distinct thresholds and cooldowns for different error types such as rate limits or quota exhaustion.
- **SQLite persistence** in [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts) maintains breaker state across process restarts.
- **Health endpoint integration** at [`src/app/api/v1/monitoring/health/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/monitoring/health/route.ts) provides real-time visibility into recovery status without additional monitoring infrastructure.

## Frequently Asked Questions

### What triggers the circuit breaker to move from OPEN to HALF_OPEN in OmniRoute?

The transition occurs when `_refreshOpenState()` detects that the effective cooldown period has elapsed during a status check. This method is invoked automatically when `canExecute()`, `getStatus()`, or `getRetryAfterMs()` are called, meaning the breaker recovers only when something actually queries its status rather than on a fixed timer.

### How does OmniRoute's circuit breaker survive application restarts?

State is persisted to a SQLite database via [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts), which stores the last failure timestamp, cumulative failure count, and error kind. When the application restarts, the breaker rehydrates its state from this table, ensuring that cooldown periods continue to respect the original failure time.

### Why doesn't OmniRoute use background timers for circuit breaker recovery?

Lazy recovery reduces CPU and memory overhead by avoiding `setInterval` or dedicated worker threads. Since `_refreshOpenState()` checks timeouts only on read operations, the system scales efficiently with thousands of provider instances without maintaining active timers for each breaker.

### What is the DEGRADED state in OmniRoute's circuit breaker?

`DEGRADED` is an intermediate state entered when failure counts exceed approximately 60% of the configured failure threshold but haven't yet reached the full opening threshold. It signals to the routing engine that the provider is struggling, enabling preemptive load reduction before the circuit fully opens and blocks all traffic.