How OmniRoute Handles Provider Fallback with Combo Routing Strategies

OmniRoute implements automatic provider fallback by iterating through ordered provider-model targets in a combo configuration, skipping targets that fail pre-flight health checks or circuit-breakers, and retrying with exponential backoff until a successful response is returned.

The OmniRoute repository provides a robust AI model routing engine designed to gracefully handle failures across 290+ providers using combo routing strategies. When processing requests, the system treats each combo as a pipeline of potential targets, evaluating candidates through independent pre-flight and runtime guards to ensure seamless failover without manual intervention.

What Are Combo Routing Strategies?

In OmniRoute, a combo is a declarative structure that groups one or more provider-model pairs with a specific routing strategy. Supported strategies include priority, weighted, round-robin, auto, and quota-share, each determining how targets are ordered and selected.

When a request arrives, the combo expands into an ordered list of ResolvedComboTarget objects. According to open-sse/services/combo/comboStructure.ts, this expansion handles wildcard resolution, fingerprint matching, and auto-candidate generation. The resulting sequence becomes the fallback chain—the exact order in which the engine attempts providers until one succeeds.

The Fallback Decision Pipeline

The core fallback orchestration resides in open-sse/services/combo.ts. Before dispatching to any target, the engine runs a series of pre-flight checks to filter out unhealthy providers. If a target passes checks but fails during execution, runtime guards trigger immediate failover.

Pre-Flight Health Checks

The following checks execute in sequence for each target in orderedTargets:

• Circuit-breaker validation: getCircuitBreaker(provider).getStatus().state must not equal "OPEN". Open circuits cause immediate skips.
• Provider-wide cooldown: isProviderInCooldown(provider, connectionId, resilienceSettings) in open-sse/services/combo/providerCooldownTracker.ts checks global cooling periods.
• Model lockout: isModelLocked(provider, connectionId, model) in open-sse/services/accountFallback.ts excludes models in quota-exhaustion lockout.
• Quota-exhaustion cutoff: resolveQuotaExhaustionCutoffForTarget in open-sse/services/combo/quotaExhaustionCutoff.ts filters connections with 0% remaining quota when enabled.
• Credential gate: checkCredentialGate(connectionId, provider, modelStr) validates API key presence and permissions.
• Custom availability hook: Optional isModelAvailable callback supplied during handleComboChat invocation for business-logic validation.
• Predictive TTFT circuit-breaker: shouldSkipForPredictedTtft preemptively skips targets predicted to exceed the predictiveTtftMs threshold.

Runtime Quality Validation

After receiving a 200 response, the engine runs validateResponseQuality from open-sse/services/combo/validateQuality.ts. If the response fails quality checks, the engine treats it as a failure and triggers fallback to the next target.

Strategy-Specific Fallback Behavior

While the fallback mechanism works uniformly across all strategies, certain optimizations apply:

Auto Strategy: When using the auto strategy defined in open-sse/services/combo/autoStrategy.ts, the engine generates candidates dynamically and applies its own cutoff logic. To avoid double-filtering, the generic fallback path skips the quota-exhaustion cutoff for auto combos.

Deterministic Strategies: For priority, weighted, and round-robin strategies, the fallback order strictly follows the original target ordering defined in comboConfig.ts. This guarantees deterministic, reproducible routing behavior even during cascading failures.

Quota-Share Strategy: This strategy implements an additional semaphore-based concurrency limit in open-sse/services/combo/quotaShareConcurrency.ts. If the per-connection slot cannot be acquired, the target is temporarily omitted from the fallback chain.

Execution Flow: From Setup to Diagnostics

The complete fallback lifecycle follows this sequence as implemented in the combo service:

1. Setup Phase: phaseComboSetup extracts the strategy, resilience settings, and metadata from the combo configuration.
2. Pre-dispatch: Handles special dispatch modes including pins, fusion, chaos, and pipeline dispatches.
3. Target Resolution: resolveComboTargetPipeline expands the combo into ResolvedComboTarget[] with wildcard and fingerprint expansion.
4. Iteration: The engine loops over orderedTargets, running the pre-flight checks. Failed checks increment a fallback counter and proceed to the next target.
5. Retry Loop: For passing targets, the engine attempts up to maxRetries with exponential backoff using retryDelayMs.
6. Execution: The request dispatches via handleSingleModelWithTimeout.
7. Post-execution: Successful responses undergo quality validation; failures trigger immediate fallback.
8. Diagnostics: If all targets fail, errorResponseWithComboDiagnostics returns detailed attempt logs, excluded providers, and retry-after hints.

Configuring Provider Fallback

You can customize fallback behavior through the combo registry and per-request overrides.

Defining a Combo with Priority Fallback

import { registerCombo } from '@/comboRegistry';

registerCombo({
  name: 'priority-fallback',
  models: [
    { provider: 'openai', model: 'gpt-4o-mini' },
    { provider: 'anthropic', model: 'claude-sonnet-4.6' },
    { provider: 'google', model: 'gemini-2.5-flash' },
  ],
  strategy: 'priority',
  config: {
    maxRetries: 2,
    retryDelayMs: 1500,
    resilienceSettings: {
      quotaPreflight: { enabled: true },
      providerCooldown: { enabled: true },
    },
  },
});

Runtime Request Overrides

When calling the OmniRoute API, override default fallback parameters:

import fetch from 'node-fetch';

const response = await fetch('https://api.omniroute.dev/v1/chat/completions', {
  method: 'POST',
  headers: { 
    'Content-Type': 'application/json', 
    'Authorization': `Bearer ${process.env.OMNIRoute_API_KEY}` 
  },
  body: JSON.stringify({
    model: 'priority-fallback',
    messages: [{ role: 'user', content: 'Explain fallback in OmniRoute.' }],
    maxRetries: 3,
    fallbackDelayMs: 500,
  }),
});

const data = await response.json();

Custom Availability Hooks

Inject custom validation logic during combo execution:

async function isModelAvailable(modelStr: string, target) {
  const connection = await getCachedProviderConnections({ 
    provider: target.provider, 
    isActive: true 
  });
  return connection.length > 0;
}

await handleComboChat({
  body: requestBody,
  combo,
  handleSingleModel,
  isModelAvailable,
  log,
  settings,
  allCombos,
  signal,
});

Advanced Fallback Features

OmniRoute includes several advanced mechanisms for optimizing fallback performance:

Shadow Routing: When a combo uses auto strategy or compatibility fallback, scheduleShadowRouting probes additional "shadow" targets in parallel, selecting the first successful response to minimize latency.

Proactive Compression: Before retrying large requests, applyCompression in open-sse/services/compression/strategySelector.ts compresses request bodies that exceed token thresholds, reducing network overhead during fallback attempts.

Compatibility Fallback: The attemptCompatRejectedFallback function in open-sse/services/combo/comboCompatFallback.ts handles legacy provider transitions, attempting alternative API formats when standard requests are rejected.

Summary

• Combo routing strategies define ordered provider chains that OmniRoute evaluates sequentially during fallback.
• Pre-flight checks in combo.ts filter targets using circuit-breakers, cooldown trackers, quota cutoffs, and credential gates before any network request.
• Runtime validation via validateResponseQuality catches degraded responses and triggers immediate failover.
• Retry logic supports configurable maxRetries and exponential retryDelayMs per combo configuration.
• Strategy-specific optimizations prevent double-filtering in auto strategies while maintaining deterministic ordering in priority and round-robin configurations.
• Advanced features like shadow routing and proactive compression optimize latency during cascading failures across the 290+ supported providers.

Frequently Asked Questions

What triggers a provider fallback in OmniRoute?

A provider fallback triggers when any pre-flight check fails—such as an open circuit-breaker, active provider cooldown, model lockout, quota exhaustion, or invalid credentials—or when a runtime request returns an error, timeout, or fails quality validation. The engine then automatically proceeds to the next target in the combo's ordered list.

How does the priority strategy differ from auto strategy in fallback behavior?

The priority strategy follows a fixed, deterministic order defined in the combo configuration, attempting providers sequentially from top to bottom. The auto strategy dynamically generates candidate targets based on real-time performance metrics and applies its own filtering logic, skipping the generic quota-exhaustion cutoff to avoid duplicate checks while potentially using shadow routing for parallel probing.

Can I implement custom logic to skip specific models during fallback?

Yes. OmniRoute accepts an optional isModelAvailable callback in handleComboChat that executes for each target during the pre-flight phase. This hook allows custom validation logic—such as checking database records for API key availability or feature flags—before the engine attempts the provider.

Where is the retry delay configured, and how does backoff work?

Retry delays are configured via the retryDelayMs parameter in the combo configuration defined in open-sse/services/combo/comboConfig.ts. The engine implements exponential backoff, increasing the delay between each retry attempt up to the configured maxRetries limit. You can override these values per request using the fallbackDelayMs field in the API request body.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}