How OmniRoute Provider Circuit Breakers Operate with Their Lazy Recovery Mechanism

OmniRoute's circuit breakers use a lazy recovery mechanism that refreshes state only when queried, eliminating background timers while ensuring providers are retried as soon as their cooldown expires.

OmniRoute protects its LLM routing pipeline from cascading failures by wrapping each provider with a circuit breaker that implements a novel lazy recovery mechanism. Unlike traditional implementations that spawn background timers to reset open circuits, OmniRoute's breaker in src/shared/utils/circuitBreaker.ts defers all state transitions until a caller explicitly checks the provider's status. This design reduces resource overhead while maintaining responsive failure recovery.

Circuit Breaker State Machine and Stages

The breaker implements a five-state machine that includes a degraded intermediate stage for early warning:

CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED

Normal Operation to Degraded

When a provider operates normally, the breaker remains in CLOSED state. As failures accumulate, the breaker enters DEGRADED once the failure count exceeds a configurable threshold—approximately 60% of the normal failure threshold according to the implementation in lines 12–18.

Open and Recovery States

Once failures cross the full threshold, _openCircuit() transitions to OPEN and a cooldown period begins. The breaker will not allow requests until this period elapses. The key innovation: no timer tracks this expiration.

Lazy Recovery: How It Works

The lazy recovery mechanism eliminates active polling or scheduled tasks. Instead, state expiration is evaluated on demand through a single private method.

The _refreshOpenState() Method

// From src/shared/utils/circuitBreaker.ts (lines 48-52)
private _refreshOpenState(): void {
  if (this._state === STATE.OPEN && this._shouldAttemptReset()) {
    this._transition(STATE.HALF_OPEN, "timeout-elapsed");
    this._halfOpenRequestsRemaining = this._halfOpenRequests;
  }
}

This method checks whether _effectiveCooldown() has elapsed only when invoked by a status query. If the timeout has passed, it atomically transitions to HALF_OPEN and initializes the probe request counter.

Entry Points That Trigger Refresh

Three public methods invoke _refreshOpenState() before returning:

Method	Purpose	Recovery Trigger
canExecute()	Permission check before routing request	Yes
getStatus()	Full state inspection for health dashboards	Yes
getRetryAfterMs()	Time-until-ready for client retry loops	Yes

This guarantees that any component querying a breaker sees fresh state without dedicated scheduling infrastructure.

Per-Kind Failure Handling and Cooldowns

OmniRoute classifies errors by kind—rate limits, quota exhaustion, authentication failures, and more. The implementation in lines 77–85 allows:

• Immediate circuit open for specific error types
• Custom cooldown durations per kind
• Escalating backoff on repeated open→probe→open cycles via _effectiveResetTimeout()

// Example: kind-specific handling in failure recording
_onFailure(error: ProviderError): void {
  const kind = this._classifyError(error);
  const threshold = this._thresholds.get(kind) ?? this._defaultThreshold;
  
  // Immediate open for quota-exhausted errors
  if (kind === 'quota-exhausted') {
    this._openCircuit({ cooldownMs: this._extendedCooldown });
    return;
  }
  
  // Standard accumulation for other failures
  this._failureCounts.set(kind, (this._failureCounts.get(kind) ?? 0) + 1);
}

The Recovery Cycle in Practice

Step-by-Step Flow

1. Failure detection — _onFailure() increments counters and may call _openCircuit()
2. Request blocking — canExecute() returns false while OPEN, with _refreshOpenState() finding cooldown incomplete
3. Elapsed cooldown detection — first query after timeout triggers _transition(STATE.HALF_OPEN, "timeout-elapsed")
4. Probe allowance — up to halfOpenRequests calls succeed through canExecute()
5. Resolution — success calls _onSuccess() to close; failure re-opens with increased backoff

// Typical pipeline integration
const breaker = getCircuitBreaker(providerName);

if (!breaker.canExecute()) {
  // Triggers _refreshOpenState() internally
  throw new CircuitBreakerOpenError(
    `Provider ${providerName} circuit open`,
    providerName,
    breaker.getRetryAfterMs(), // Also triggers refresh
  );
}

// Execute actual LLM call
const response = await llmClient.complete(request);
breaker.recordSuccess(); // Internal _onSuccess() handles transition

Persistence and Observability

SQLite State Survival

Breaker state persists to src/lib/db/domainState.ts, enabling:

• Process restart recovery — open circuits remain open across deployments
• Historical analysis — failure patterns visible in domain-state table
• Cross-instance coordination — single source of truth for multi-worker deployments

Health Endpoint Integration

The monitoring route at src/app/api/v1/monitoring/health/route.ts exposes real-time breaker status:

// Health dashboard integration
app.get("/api/v1/monitoring/health", async (req, res) => {
  const providers = Object.keys(providerRegistry);
  const breakerStatuses = providers.map((p) => 
    getCircuitBreaker(p).getStatus()  // Forces refresh on each call
  );
  res.json({ providers: breakerStatuses });
});

Operators observe lazy recovery in real time: retryAfterMs counts down only when the endpoint is polled, and state transitions appear on the next query after cooldown expiration.

Comparison: Lazy vs. Eager Recovery

Approach	Implementation	Resource Cost	Recovery Latency
Lazy (OmniRoute)	Refresh on read	Zero idle overhead	Zero to one polling interval
Eager (traditional)	Background timer/setTimeout	Per-breaker timer	Immediate on expiration

OmniRoute's lazy approach suits serverless and edge environments where background tasks may be frozen or throttled. The tradeoff—slightly delayed recovery if no component queries the breaker—is mitigated by omnipresent health checks and active routing engines.

Summary

• OmniRoute circuit breakers implement lazy recovery through _refreshOpenState() invoked only on status queries
• No background timers run; state transitions occur in canExecute(), getStatus(), or getRetryAfterMs()
• The five-state machine includes DEGRADED for early intervention before full circuit open
• Per-kind error handling enables differentiated thresholds and cooldowns
• SQLite persistence in domainState.ts ensures survival across restarts
• Health endpoint integration provides observable, on-demand recovery status

Frequently Asked Questions

What triggers a circuit breaker to move from OPEN to HALF_OPEN?

The transition occurs when any code path calls a method that invokes _refreshOpenState()—typically canExecute(), getStatus(), or getRetryAfterMs()—and the elapsed time since opening exceeds _effectiveCooldown(). No timer or scheduled task performs this check; it happens exclusively on demand.

How does lazy recovery affect provider availability after cooldown?

Recovery latency depends on query frequency. In practice, OmniRoute's routing engine checks canExecute() before every request attempt, and health endpoints poll getStatus() regularly. This ensures providers become eligible for traffic within one polling interval of cooldown expiration without dedicated timer infrastructure.

Can circuit breaker state survive application restarts?

Yes. The implementation persists state to an SQLite table via src/lib/db/domainState.ts, including last failure time, failure counts by kind, and current state. On initialization, breakers reload this state and immediately apply lazy recovery logic—an open circuit with expired cooldown will transition to HALF_OPEN on its first query.

What happens when probe requests fail in HALF_OPEN state?

Failure during the halfOpenRequests probe window immediately re-opens the circuit. The _onFailure() handler applies escalating backoff through _effectiveResetTimeout(), which increases duration based on the count of open→probe→open cycles. This prevents rapid oscillation on persistently unhealthy providers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}