How OmniRoute's Quota-Share Routing Distributes Shared Account Quotas Across Pooled Keys

OmniRoute uses a three-phase, in-process algorithm—bucket gating, Deficit Round-Robin ordering, and Power-of-Two-Choices load balancing—to fairly distribute shared account quotas across pooled API keys while remaining fail-open.

OmniRoute's quota-share routing is designed for combos auto-minted with the qtSd/ prefix. It runs entirely in-process with no database or network calls, ensuring minimal latency on every request. This routing strategy balances fairness, quota protection, and real-time load awareness through a deterministic, multi-phase selection pipeline.

The Three-Phase Quota Distribution Algorithm

Quota-share routing processes every request through three ordered phases. Each phase refines the candidate pool while maintaining a fail-open guarantee—requests never hard-block even when all connections are saturated.

Phase 1: Per-Model Bucket Gating

First, OmniRoute inspects each connection's usage buckets—tracking 5-hour, 7-day, and per-model 7-day windows. Connections with saturated buckets are de-prioritized to the tail of the candidate list but never removed entirely.

In open-sse/services/combo/quotaShareStrategy.ts, the filterEligibleBySaturation function implements this logic:

// From quotaShareStrategy.ts
function filterEligibleBySaturation(
  targets: ResolvedComboTarget[],
  modelRef: string,
  now: number
): { eligible: ResolvedComboTarget[]; saturated: ResolvedComboTarget[] } {
  // Uses isBucketSaturated from src/lib/quota/accountBuckets.ts
}

The isBucketSaturated helper in src/lib/quota/accountBuckets.ts performs the actual bucket threshold checks against configured quota limits.

Phase 2: Deficit Round-Robin (DRR) Ordering

On the remaining eligible connections, OmniRoute applies Deficit Round-Robin to achieve weight-proportional distribution. The algorithm maintains a deficit counter per target:

• Each round adds a quantum proportional to the target's weight
• The target with the highest deficit is selected
• Its deficit is decremented by 1

Over many requests, this yields deterministic, fair allocation according to configured weights.

The DRR state persists in an internal _drrState Map keyed by combo name. The applyDrr function in quotaShareStrategy.ts manages this state:

// DRR state maintenance from quotaShareStrategy.ts
private _drrState = new Map<string, Map<string, number>>(); // combo → targetId → deficit

function applyDrr(
  targets: ResolvedComboTarget[],
  comboKey: string,
  now: number
): ResolvedComboTarget[] {
  // Initialize or retrieve deficits, add quantums, sort by deficit descending
}

Phase 3: Power-of-Two-Choices Load Balancing

Finally, OmniRoute selects between the top two DRR candidates using Power-of-Two-Choices (P2C) over real-time in-flight load:

1. Compare the two candidates by current in-flight request count
2. The lighter-loaded target wins; ties preserve the DRR ordering
3. The chosen target's counter increments immediately
4. A decrementInflight callback is returned for cleanup

This happens in open-sse/services/combo/quotaShareInflight.ts:

// From quotaShareInflight.ts
function pickByInflightP2C(candidates: ResolvedComboTarget[]): ResolvedComboTarget {
  const [first, second] = candidates;
  const firstLoad = getInflightCount(first.connectionId);
  const secondLoad = second ? getInflightCount(second.connectionId) : Infinity;
  
  return firstLoad <= secondLoad ? first : second;
}

function incrementInflight(connectionId: string): () => void {
  // Atomically increment and return decrement callback
}

Optional: Per-Connection Concurrency Cap Protection

Before DRR processing, connections with maxConcurrent limits (from provider_connections.max_concurrent) are evaluated. Connections at their cap are de-prioritized behind those with headroom, implemented in partitionByConcurrencyCap within quotaShareStrategy.ts.

This step also respects the fail-open principle—at-cap connections remain in the pool, just ordered later.

Final Dispatch Order

The complete selection priority for quota-share routing is:

1. Winner — selected by P2C from top DRR candidates
2. Remaining with headroom — eligible, non-saturated, under cap
3. At-cap connections — concurrency-limited but available
4. Saturated connections — de-prioritized but never dropped

winner (P2C) → eligible with headroom → at-cap → saturated

Using Quota-Share Routing Programmatically

OmniRoute exposes selectQuotaShareTarget for custom executors or testing:

import { selectQuotaShareTarget } from "@/open-sse/services/combo/quotaShareStrategy";
import type { ResolvedComboTarget } from "@/open-sse/services/combo/types";

const targets: ResolvedComboTarget[] = [
  { connectionId: "c1", executionKey: "c1-0", weight: 2, provider: "anthropic", model: "claude-opus-4" },
  { connectionId: "c2", executionKey: "c2-0", weight: 1, provider: "openai", model: "gpt-4" },
];

const caps = new Map<string, number | null>([
  ["c1", 5],
  ["c2", null], // null or ≤0 means no limit
]);

const result = selectQuotaShareTarget(
  targets,
  "qtSd/my-shared-combo",      // combo name serves as DRR state key
  "anthropic/claude-opus-4",   // requested model for bucket gating
  Date.now(),
  { maxConcurrentByConnection: caps }
);

console.log("Chosen target:", result.target);
console.log("Ordered candidates:", result.orderedTargets);

// Always release the in-flight slot when done
try {
  // ...execute request against result.target...
} finally {
  result.decrementInflight(); // idempotent, safe if missed
}

Deterministic Testing

For unit tests, clear DRR state and use fixed timestamps:

import { selectQuotaShareTarget, _clearDrrStateForTest } from "@/open-sse/services/combo/quotaShareStrategy";

_clearDrrStateForTest();
const now = 1722836400000; // fixed epoch ms

const { target } = selectQuotaShareTarget(targets, "combo-test", "openai/gpt-4", now);
expect(target?.connectionId).toBe("c1"); // deterministic outcome

Key Implementation Files

File	Purpose
open-sse/services/combo/quotaShareStrategy.ts	Core three-phase algorithm (bucket gating → DRR → P2C)
src/lib/quota/accountBuckets.ts	isBucketSaturated for quota window checking
open-sse/services/combo/quotaShareInflight.ts	In-flight tracking and P2C selection logic
open-sse/services/combo/quotaShareConcurrency.ts	Per-connection concurrency semaphore
tests/unit/quota-share-strategy.test.ts	Unit tests verifying fail-open behavior and algorithm correctness

Summary

• Quota-share routing in diegosouzapw/OmniRoute handles qtSd/ prefixed combos with a three-phase, in-process algorithm
• Bucket gating de-prioritizes saturated connections using 5h, 7d, and per-model 7d windows from accountBuckets.ts
• Deficit Round-Robin ensures weight-proportional distribution via maintained deficit counters in _drrState
• Power-of-Two-Choices balances real-time load across the top two DRR candidates using in-flight counters
• Fail-open design guarantees every request receives a target, even when all connections are constrained

Frequently Asked Questions

How does OmniRoute prevent quota exhaustion with quota-share routing?

OmniRoute tracks per-connection usage in three time windows (5-hour, 7-day, and per-model 7-day) via isBucketSaturated in src/lib/quota/accountBuckets.ts. Saturated connections are de-prioritized but remain available—this fail-open approach prevents hard failures while steering traffic toward healthier keys.

What makes Deficit Round-Robin better than simple weighted random selection?

Deficit Round-Robin provides deterministic, perfectly weight-proportional distribution over time. Unlike random approaches that exhibit variance, DRR guarantees that a weight-2 target receives exactly twice the requests of a weight-1 target across any sufficiently large window. The deficit counters in _drrState make this efficient and stateful without external storage.

Can quota-share routing handle connections with different concurrency limits?

Yes. The optional partitionByConcurrencyCap step in quotaShareStrategy.ts evaluates provider_connections.max_concurrent values. Connections at their limit are ordered after those with available capacity, while the fail-open principle ensures they remain as fallback candidates.

Why use Power-of-Two-Choices instead of selecting the single best DRR candidate?

Power-of-Two-Choices combines DRR's fairness guarantee with real-time load awareness. By comparing only the top two DRR-scored candidates by in-flight count, OmniRoute avoids thundering-herd problems and hotspotting while maintaining O(1) selection complexity. The in-flight tracking in quotaShareInflight.ts provides immediate visibility into actual load pressure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}