# How OmniRoute's Key Pooling Manages API Key Quotas

> OmniRoute's key pooling efficiently manages API key quotas by distributing requests, automatically sidelining exhausted keys, and spreading consumption evenly across your key set.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-09

---

**OmniRoute's key pooling distributes requests across multiple API credentials in a connection pool, automatically sidelining exhausted keys via cooldown timestamps and exponential backoff while spreading quota consumption evenly across the key set.**

OmniRoute (diegosouzapw/OmniRoute) implements a sophisticated **key pooling** mechanism that treats every provider credential as an individual connection within a managed pool. Instead of routing traffic to a single API key until failure, the system proactively distributes load across available credentials while maintaining per-key state to enforce quota limits. This architecture prevents any single key from becoming a bottleneck and enables automatic recovery when rate limits reset.

## The Connection Pool Architecture

Every API key, OAuth token, or service account registered in OmniRoute exists as a distinct **connection** object within the pool. This abstraction allows the router to apply granular control policies to individual credentials rather than treating provider access as a monolithic resource.

The system tracks three critical pieces of state for each connection to manage quota exhaustion:

- **`rateLimitedUntil`**: A timestamp indicating when a connection becomes eligible for selection again, enforced after quota exhaustion or HTTP 429 responses
- **`backoffLevel`**: An exponential-backoff counter that determines cooldown duration using the formula `baseCooldownMs * 2 ** backoffLevel`
- **`testStatus` / `errorCode`**: Classification of the last failure to distinguish transient quota errors from permanent authentication failures

## State Management and Cooldown Logic

### Marking Accounts Unavailable

In [`src/open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountFallback.ts), the `markAccountUnavailable` function implements the core quarantine logic. When a key returns a quota-exceeded error, this function sets the `rateLimitedUntil` timestamp and increments the `backoffLevel`, calculating the next interval as `baseCooldownMs * 2 ** backoffLevel`.

The companion function `clearAccountError` resets these states when a connection proves healthy again, allowing the key to re-enter the rotation without manual intervention.

### Error Classification

The same file maintains `testStatus` and `errorCode` fields to distinguish between temporary quota exhaustion (which triggers automatic cooldown) and permanent failures such as revoked credentials (which require administrative action).

## Connection Selection Strategy

### Filtering and Ranking Algorithm

During request routing, [`src/open-sse/services/accountSelector.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountSelector.ts) executes a three-phase selection process:

1. **Availability Filter**: Eliminates any connection where `rateLimitedUntil` is in the future
2. **Usage Ranking**: Sorts remaining connections by a least-used heuristic based on `percentUsed` statistics from [`src/open-sse/services/quotaMonitor.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/quotaMonitor.ts)
3. **Health Verification**: Validates the selected connection before dispatching the request

### Quota-Aware Fallback Handling

When a selected key returns a quota-related error (HTTP 429/403 with "quota exceeded"), [`src/open-sse/services/quotaPreflight.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/quotaPreflight.ts) and [`src/open-sse/services/quotaFetchThrottle.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/quotaFetchThrottle.ts) coordinate the fallback mechanism. These services interpret upstream error codes and invoke `markAccountUnavailable` to sideline the exhausted key while immediately selecting an alternative from the pool.

## Provider-Level Circuit Breaking

Beyond individual key management, [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) implements provider-wide protection. If multiple keys for a single provider repeatedly fail within a short window, the circuit breaker opens and skips the entire provider until the failure rate subsides. This prevents cascading failures where a provider-side outage exhausts all keys in rapid succession, working in concert with the per-key pooling strategy.

## Implementation Examples

```typescript
// Selecting a ready connection from the pool
import { selectConnection } from "@/open-sse/services/accountSelector";

const conn = await selectConnection({
  provider: "openai",
  model: "gpt-4o",
}); // Returns a connection whose key is not in cooldown

```

```typescript
// Handling quota exhaustion with backoff
import { markAccountUnavailable } from "@/open-sse/services/accountFallback";

if (response.status === 429 && response.body?.error?.includes("quota")) {
  await markAccountUnavailable(connectionId, {
    backoffMs: 30_000, // 30s cooldown
    reason: "quota_exhausted",
  });
}

```

## Summary

- **Connection Pool Model**: Each API credential exists as an independent connection with isolated state tracking in [`src/open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountFallback.ts)
- **Automatic Cooldown**: The `rateLimitedUntil` timestamp and `backoffLevel` counter implement exponential backoff for exhausted keys
- **Intelligent Selection**: [`src/open-sse/services/accountSelector.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountSelector.ts) filters unavailable keys and ranks by usage statistics from [`src/open-sse/services/quotaMonitor.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/quotaMonitor.ts)
- **Graceful Degradation**: Quota errors trigger immediate fallback to healthy keys via [`src/open-sse/services/quotaPreflight.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/quotaPreflight.ts) and [`src/open-sse/services/quotaFetchThrottle.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/quotaFetchThrottle.ts)
- **Circuit Protection**: Provider-level breakers in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) prevent total provider failure when multiple keys hit limits simultaneously

## Frequently Asked Questions

### How does OmniRoute prevent a single API key from consuming the entire quota?

OmniRoute's key pooling spreads requests across all available credentials for a provider. The `selectConnection` function in [`src/open-sse/services/accountSelector.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountSelector.ts) ranks connections by least-recent usage statistics gathered from [`src/open-sse/services/quotaMonitor.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/quotaMonitor.ts), ensuring even distribution rather than sequential exhaustion of individual keys.

### What happens when all API keys in the pool hit their rate limits?

If all keys enter cooldown simultaneously, the provider-level circuit breaker in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) opens, temporarily halting requests to that provider. Once the earliest `rateLimitedUntil` timestamps expire and keys become available, or the breaker half-opens, traffic resumes automatically without manual intervention.

### How long does OmniRoute wait before retrying an exhausted API key?

The system uses exponential backoff calculated as `baseCooldownMs * 2 ** backoffLevel` within `markAccountUnavailable` in [`src/open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountFallback.ts). The `backoffLevel` increments with each consecutive failure, extending the `rateLimitedUntil` timestamp dynamically based on repeated quota exhaustion patterns.

### Can OmniRoute distinguish between temporary quota limits and permanently revoked keys?

Yes. The `testStatus` and `errorCode` fields tracked in [`src/open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountFallback.ts) classify failures differently. Transient quota errors trigger temporary cooldowns, while authentication failures or permanent revocations are flagged separately, preventing wasted retries on invalid credentials.