How OmniRoute Combo Routing Handles Fallback with 19 Strategies

OmniRoute delegates all 19 routing strategies to a shared fallback pipeline that automatically retries exhausted providers, rebuilds fallback tiers, and triggers last-resort compatibility modes to ensure every LLM request succeeds or fails gracefully with a structured error response.

OmniRoute is an open-source LLM routing engine developed by diegosouzapw that intelligently distributes requests across multiple providers. Its combo routing system supports 19 distinct strategies—from priority and weighted to context-optimized and fusion—all of which rely on a centralized fallback framework to ensure high availability when primary targets fail. Understanding how OmniRoute combo routing handles fallback is essential for building resilient AI applications that automatically degrade to alternative models without manual intervention.

Unified Fallback Architecture

All 19 strategies delegate target selection to a shared fallback pipeline implemented across comboStructure.ts, targetExhaustion.ts, and comboCompatFallback.ts. This architecture ensures consistent behavior whether you are using simple round-robin or complex auto-combo scoring.

Candidate Generation and Compatibility Filtering

Each strategy begins by producing an ordered list of ResolvedComboTarget objects. Before any target is attempted, filterTargetsByRequestCompatibility removes candidates that are outright incompatible with the request features, such as models lacking required capabilities.

Fallback Tier Construction

The comboStructure.ts module builds a hierarchy of fallback tiers. The primary tier contains the strategy’s top-ranked targets, while subsequent tiers represent lower-priority, last-resort candidates. This tiered structure allows the router to exhaust all viable options at one priority level before descending to the next.

Exhaustion and Cool-Down Checks

Before attempting a target, targetExhaustion.ts and providerCooldownTracker.ts inspect whether the provider is exhausted due to quota limits, rate limiting, or circuit-breaker states. If a target is exhausted, the fallback pointer advances to the next candidate in the tier without returning an error to the client.

Retry-After Handling

When transient errors occur (HTTP 429 or 503), comboPredicates.ts and unavailableRetryGate.ts compute a retry-after delay. If this delay is below MAX_FALLBACK_WAIT_MS, the router pauses execution and retries the same target; otherwise, it proceeds to the next fallback tier immediately.

Last-Resort Compatibility Fallback

If every candidate in the primary tier is rejected due to compatibility constraints, comboCompatFallback.ts constructs a last-resort compat fallback tier. As noted in the source logs (Last-resort compat fallback), this tier activates only after all higher-priority tiers have been exhausted, attempting the next best-compatible models to prevent request failure.

Result Aggregation

Upon success, the router calls recordProviderSuccess to update provider health metrics and returns the response. If all tiers fail, the pipeline generates a comboErrorResponse containing the aggregated failure reasons, ensuring the client receives actionable error data.

How the 19 Strategies Use the Fallback Pipeline

Every strategy invokes the same fallback engine, but each generates its initial candidate list differently. The following patterns demonstrate how specific strategies integrate with the shared fallback framework:

• Priority: Selects the highest-priority target first; if it fails, the fallback routine moves to the next priority level in the tier hierarchy.
• Weighted: Orders targets by weight; the fallback tier respects this weighted order while skipping exhausted or throttled connections.
• Round-Robin: Maintains per-provider counters (rrCounters); when a target is exhausted, the counter advances to the next candidate in the fallback tier.
• Random: Generates a randomized ordering once per request; fallback follows this same random list to ensure fair distribution of retries.
• Least-Used: Sorts targets by recent usage statistics; fallback proceeds down the list to prevent over-use of hot connections.
• Cost-Optimized: Ranks targets by cost-per-token; fallback respects cost ordering while avoiding blocked connections.
• Reset-Aware / Reset-Window: Uses RESET_WINDOW_NAMES to group targets; if a reset window expires, the fallback tier may be re-evaluated with fresh candidates.
• Strict-Random: Guarantees a fresh random order for each request; fallback continues with the same randomized sequence.
• Auto: Auto-combo scores candidates and iteratively re-ranks them; each iteration is treated as a separate fallback tier (see resolveAutoStrategy.ts).
• Fill-First: Fills the request with the first available target; if unavailable, fallback proceeds to the next fill candidate.
• P2C (Power-of-Two-Choices): Picks two random candidates and selects the healthier one; if both are exhausted, the fallback tier supplies the next pair.
• LKGP (Least-Known-Good-Performance): Selects the target with historically lowest latency; fallback works identically if that target is throttled.
• Context-Optimized / Context-Relay: Considers token-budget and context-overflow patterns; if the chosen target would overflow, the fallback tier supplies a target with higher token capacity.
• Fusion: Fans out to a panel of models and merges results; each panel member is still subject to the same fallback checks.
• Quota-Share: Shares quota across accounts; if a quota share slot is unavailable, quotaShareStrategy.ts falls back to the next eligible target.
• Prompt-Cache-Affinity: Prioritizes targets with cached prompt data; if cache-affinity fails, fallback reverts to the generic tier.
• Pin-Recovery: Handles pinned-model failures; on pin failure the router explicitly falls back to the next tier (see pinRecovery.ts).

Code Examples

Simple Priority Routing with Fallback

The following pattern demonstrates how priority-based routing automatically triggers fallback when the primary provider returns a 429 or enters cooldown:

// Request handling (simplified)
const targets = resolveComboTargets({
  strategy: 'priority',
  // …request‑specific config…
});
const result = await handleComboRequest(targets);

If the top-priority provider returns a 429 or is in cooldown, handleComboRequest will:

1. Detect the error via isAllAccountsRateLimitedResponse.
2. Increment fallbackCount (tracked in runtimeUnits.ts).
3. Move to the next fallback tier defined in comboStructure.ts.
4. Return the successful response or an error after exhausting all tiers.

Auto-Combo with Iterative Fallback

The auto strategy re-scores candidates on each iteration, dynamically rebuilding fallback tiers:

const autoTargets = resolveComboTargets({
  strategy: 'auto',
  // auto‑combo scoring config
});
await resolveAutoStrategy(autoTargets); // internally rebuilds fallback tiers

Each scoring iteration (see resolveAutoStrategy.ts) may de-prioritize previously selected targets. The fallback tier is recomputed after each pass, allowing the router to gracefully degrade to lower-scored providers when higher-scored ones become unavailable.

Last-Resort Compatibility Fallback

When all primary targets fail compatibility checks, the router constructs a final fallback tier:

// Inside comboCompatFallback.ts
if (allPrimaryTargetsFailed) {
  ctx.log.info('COMBO', `Last-resort compat fallback → ${target.modelStr}`);
  // The fallback tier is constructed here and retried.
}

This ensures that even when every primary target is filtered out due to incompatibility, the request still has a chance to succeed using the next best-compatible models.

Key Source Files

According to the diegosouzapw/OmniRoute source code, the following files implement the strategy-agnostic fallback mechanism:

• combo.ts: Entry point that wires all 19 strategies together and imports shared fallback utilities.
• comboStructure.ts: Builds hierarchical fallback tiers and normalizes target lists.
• comboCompatFallback.ts: Implements the last-resort compatibility fallback tier.
• targetExhaustion.ts: Detects exhausted, rate-limited, or circuit-broken targets.
• comboPredicates.ts: Provides predicate functions like isAllAccountsRateLimitedResponse that trigger fallback decisions.
• comboCooldownRetry.ts: Handles cooldown-aware retries before falling back.
• targetResolution.ts: Refines fallback order without overriding primary routing decisions.
• resolveAutoStrategy.ts: Shows how auto-combo re-scores candidates and rebuilds fallback tiers.
• quotaShareStrategy.ts: Demonstrates quota-share fallback handling.
• promptCacheAffinity.ts: Illustrates fallback when prompt-cache affinity cannot be satisfied.
• pinRecovery.ts: Handles explicit fallback for pinned-model failures.

Summary

• OmniRoute’s combo router unifies 19 distinct strategies under a single fallback pipeline defined in comboStructure.ts and targetExhaustion.ts.
• The fallback system constructs tiered candidate lists, filters exhausted providers via providerCooldownTracker.ts, and handles transient errors with configurable MAX_FALLBACK_WAIT_MS delays.
• When all primary targets fail, comboCompatFallback.ts activates a last-resort compatibility tier to prevent request failure.
• Every strategy—from simple round-robin to complex fusion panels—inherits identical fallback behavior without code duplication.

Frequently Asked Questions

What happens when all 19 strategies exhaust their fallback tiers?

If every candidate across all fallback tiers is exhausted or incompatible, the router returns a comboErrorResponse containing aggregated failure details. This structured error includes information about which providers were attempted and why they failed (rate limits, circuit breakers, or incompatibility), allowing clients to implement their own recovery logic.

How does OmniRoute handle rate limiting during fallback?

When a provider returns a 429 status, comboPredicates.ts detects the condition via isAllAccountsRateLimitedResponse. The router then consults unavailableRetryGate.ts to calculate a retry-after delay. If the delay is below MAX_FALLBACK_WAIT_MS, the request pauses and retries the same target; otherwise, it advances to the next fallback tier immediately.

Can the auto strategy reuse previously skipped targets in later fallback tiers?

Yes. The auto strategy, implemented in resolveAutoStrategy.ts, re-scores candidates on each iteration. Because the fallback tier is rebuilt after every scoring pass, targets that were initially skipped due to transient high load can re-enter the candidate pool in later iterations, enabling dynamic recovery and graceful degradation.

Does the fallback mechanism work differently for fusion versus priority routing?

No. While fusion fans out to multiple models simultaneously and priority selects a single ordered target, both strategies use the same underlying fallback engine. Each target within a fusion panel is individually subject to exhaustion checks in targetExhaustion.ts and compatibility filtering, ensuring consistent reliability across all 19 routing strategies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}