# How OmniRoute's Circuit Breaker Differentiates Provider-Level vs Account-Level Errors

> Learn how OmniRoute's circuit breaker distinguishes provider-level from account-level errors, enhancing your system's resilience against failures.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-20

---

**OmniRoute uses a two-layer resilience system: provider-level circuit breakers block entire upstream providers when systemic failures occur, while account-level cooldowns isolate individual credentials when only one API key or token fails.**

The `diegosouzapw/OmniRoute` routing engine implements granular fault tolerance that prevents a single bad credential from taking down an entire provider, while also protecting against thundering-herd scenarios during upstream outages. This dual-layer approach lives primarily in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts).

## Provider-Level Circuit Breakers: Protecting Against Systemic Failures

OmniRoute aggregates failures across **all connections to a provider** using the `CircuitBreaker` class. When failure rates exceed configured thresholds, the entire provider is temporarily blocked.

### How Provider Breakers Are Triggered

In [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), errors are classified by HTTP status code. Only failures matching `PROVIDER_FAILURE_ERROR_CODES` (defined at lines 226-227) propagate to the provider-level breaker:

```

408 (Request Timeout), 429 (Rate Limit), and 500-504 (Server Errors)

```

These indicate infrastructure problems rather than credential-specific issues.

```typescript
// From open-sse/services/accountFallback.ts
const PROVIDER_FAILURE_ERROR_CODES = [408, 429, 500, 501, 502, 503, 504];

```

The breaker singleton is retrieved via `getCircuitBreaker(provider)` and updated through `CircuitBreaker.execute()` in the request pipeline.

### Deduplication Prevents Inflation

To avoid skewing provider breaker metrics from rapid retries on the same failing connection, OmniRoute maintains two deduplication windows:

- **`lastConnectionFailure`**: 5-second window per connection
- **`lastNetworkErrorByProvider`**: 10-second window per provider

Implemented at lines 28-41 of [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts), these ensure that a single network blip doesn't immediately open the provider breaker, while still capturing genuinely distributed failures.

### State Transitions in circuitBreaker.ts

The `CircuitBreaker` class manages three states:

- **`CLOSED`**: Normal operation, failures tracked
- **`OPEN`**: All requests short-circuited, `canExecute()` returns `false`
- **`HALF_OPEN`**: Testing recovery with limited traffic

State transitions occur in `_onFailure(kind)` (lines 94-106 of [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts)). When `failureThreshold` is exceeded, the breaker moves to `OPEN` for `resetTimeout` milliseconds.

## Account-Level Cooldowns: Isolating Individual Credentials

When an error **doesn't** indicate systemic provider failure—such as `401 Unauthorized`, `403 Forbidden`, or model-specific quota exhaustion—OmniRoute handles it at the connection level.

### Connection State Tracking

Per-account resilience is configured through `ProviderProfile` fields in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) (lines 20-30):

| Field | Purpose |
|-------|---------|
| `baseCooldownMs` | Initial backoff duration for this account |
| `maxBackoffSteps` | Maximum exponential backoff iterations |
| `rateLimitCooldown` | Specific duration for 429 responses |

### The Account Availability Pipeline

1. **Error classification**: Functions like `classify429FromError` categorize failures
2. **State recording**: `markAccountUnavailable` writes `rateLimitedUntil`, `backoffLevel`, and `lastError` to the database
3. **Execution check**: `canExecute` in request handlers queries `isAccountUnavailable` (lines 41-44) to skip cooled connections
4. **Recovery**: `recordProviderSuccess` clears cooldowns when requests succeed

```typescript
// Conceptual flow from accountFallback.ts
if (errorStatus === 429 && error.code === 'insufficient_quota') {
  // Account-level: only this API key is affected
  await markAccountUnavailable(connectionId, 'quota_exhausted', backoffMs);
} else if (PROVIDER_FAILURE_ERROR_CODES.includes(errorStatus)) {
  // Provider-level: contributes to aggregate circuit breaker
  getCircuitBreaker(provider).recordFailure();
}

```

## Decision Flow: Where Errors Get Routed

```

Incoming Error
    │
    ▼
┌─────────────────┐
│ HTTP Status Code│
│   Classification │
└────────┬────────┘
         │
    ┌────┴────┐
    ▼         ▼
408,429,5xx  401,403,quota
(PROVIDER)   (ACCOUNT)
    │         │
    ▼         ▼
┌─────────┐ ┌─────────────────┐
│Deduplicate│ │markAccountUnavailable│
│  (5s/10s) │ │  update DB row       │
└────┬────┘ └─────────────────┘
     │
     ▼
┌─────────────────┐
│getCircuitBreaker│
│  (provider)     │
│  .recordFailure()│
└────────┬────────┘
         ▼
    Threshold Check
    (failureThreshold)
         │
    ┌────┴────┐
    ▼         ▼
   CLOSED    THRESHOLD EXCEEDED
              │
              ▼
            OPEN
   (all provider requests blocked)

```

## Configuration and Operational Controls

### Inspecting Breaker Status

```typescript
import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";

const cb = getCircuitBreaker("openai");
console.log(cb.getStatus());
// {
//   name: "openai",
//   state: "CLOSED",
//   failureCount: 0,
//   retryAfterMs: 0,
//   ...
// }

```

### Checking Individual Account Availability

```typescript
import { getProviderCredentials } from "@omniroute/open-sse/services/auth";

const creds = await getProviderCredentials("openai", null, null, null, {
  allowRateLimitedConnections: false,
});
if (!creds) {
  console.log("All OpenAI accounts are currently rate‑limited or unavailable");
}

```

### Emergency Reset

```typescript
import { resetAllCircuitBreakers } from "@/shared/utils/circuitBreaker";

resetAllCircuitBreakers(); // Clears provider-level state post-deployment

```

## Key Source Files

| File | Responsibility | Link |
|------|---------------|------|
| [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | Generic breaker implementation, provider registry, persistence | [circuitBreaker.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/shared/utils/circuitBreaker.ts) |
| [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Error classification, deduplication, account cooldown logic | [accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts) |
| [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) | Default breaker profiles (`failureThreshold`, `resetTimeout`) | [settings.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/lib/resilience/settings.ts) |
| [`src/lib/monitoring/providerHealthMatrix.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/monitoring/providerHealthMatrix.ts) | Health API exposure of breaker states | [providerHealthMatrix.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/lib/monitoring/providerHealthMatrix.ts) |

## Summary

- **Provider-level circuit breakers** in [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts) protect against upstream infrastructure failures using aggregated failure counts across all connections, triggered by HTTP 408/429/5xx status codes with 5-10 second deduplication windows.

- **Account-level cooldowns** in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) isolate individual credentials through database-tracked `rateLimitedUntil` timestamps and exponential backoff, preserving healthy accounts when one API key fails.

- **Dual-layer classification** happens at error time: systemic codes hit the provider breaker, credential-specific issues trigger per-account throttling—ensuring neither noisy neighbors nor upstream outages degrade routing availability.

## Frequently Asked Questions

### How do I know if a provider is in OPEN state versus just having rate-limited accounts?

Check the provider breaker directly via `getCircuitBreaker(provider).getStatus()`. When `state` equals `"OPEN"`, all accounts are blocked. If `state` is `"CLOSED"` but requests still fail, individual accounts likely have active cooldowns—query `isAccountUnavailable` for specific connection IDs.

### Can I adjust the failure threshold for a specific provider?

Yes. Pass a custom profile when initializing the breaker: `getCircuitBreaker("anthropic", { failureThreshold: 5, resetTimeout: 30000 })`. The default profiles are defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) and applied at [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) lines 84-89.

### Why does OmniRoute deduplicate provider failures instead of counting every error?

The 5-second `lastConnectionFailure` and 10-second `lastNetworkErrorByProvider` windows prevent scenario where rapid retries on one bad connection artificially inflate the aggregate failure rate. This ensures the provider breaker genuinely reflects distributed outages, not localized retry storms.

### What happens when a provider breaker is OPEN but some accounts still have quota?

All traffic to that provider is rejected regardless of individual account health. This is intentional: if OpenAI returns 503s across multiple accounts, attempting alternate credentials wastes latency and may exacerbate the upstream degradation. Accounts resume eligibility once the breaker transitions to `HALF_OPEN` after `resetTimeout`.