How OmniRoute Handles Routing: Inside the Combo Routing Engine

OmniRoute handles routing through its Combo Routing Engine, which selects optimal upstream providers using 17 configurable strategies—including weighted, cost-optimized, and context-aware—by iterating through resolved targets in open-sse/services/combo.ts until a successful response is obtained or emergency fallback is triggered.

OmniRoute is an open-source LLM gateway that optimizes request distribution across multiple AI providers. Understanding how OmniRoute handles routing is essential for operating large-scale AI workloads that demand high availability and cost efficiency. The system's routing logic centers on combos—configurable provider sequences that balance latency, cost, and reliability through dynamic strategy selection and automatic failover.

The Combo-Centric Routing Architecture

OmniRoute’s routing optimization revolves around the Combo Routing Engine, which treats each request as a configurable workflow rather than a single endpoint call. This architecture separates routing into distinct phases: definition, resolution, strategy application, and execution.

Combo Definition and Storage

A combo is a configuration object stored in the SQLite database (src/lib/db/combo.ts) that lists one or more provider-model pairs. Each combo supports optional weighting, fallback rules, and priority settings. Combos can be created through the admin UI or CLI, making routing logic transparent and version-controlled.

Target Resolution

When a request arrives, open‑sse/services/combo.ts invokes resolveComboTargets(). This function expands the combo definition into an ordered array of ResolvedComboTarget objects. Each target contains the concrete provider ID, model name, account credentials (resolved from encrypted DB fields via src/lib/db/encryption.ts), and per-target overrides.

The 17 Routing Strategies Explained

OmniRoute supports 17 routing strategies defined in src/shared/constants/routingStrategies.ts. The chosen strategy determines how resolved targets are traversed:

• Priority: Sequential failover through ordered targets.
• Weighted: Distribution based on configured percentages (e.g., 70% OpenAI, 30% Groq).
• Round-robin: Cyclical distribution across targets.
• P2C (Power of Two Choices): Selects between two randomly chosen targets based on load.
• Cost-optimized: Selects the cheapest viable provider using live data from src/lib/pricingSync.ts.
• Context-optimized: Routes based on token consumption headroom and context window availability.

Other strategies include random selection, latency-optimized, and various hybrid approaches. The strategy identifier is attached to the combo configuration and evaluated during the execution phase.

Execution Flow: From Request to Response

The actual routing execution follows a strict pipeline that ensures reliability and real-time optimization.

The Combo Execution Loop

handleComboChat() in open‑sse/services/combo.ts implements the core execution loop. For each resolved target, it invokes handleSingleModel(), which wraps the standard chat handling pipeline from open‑sse/handlers/chatCore.ts with combo-specific logic. The loop includes circuit-breaker checks and error handling. If a target succeeds, the response returns immediately; otherwise, the loop proceeds to the next target according to the selected strategy until the list is exhausted.

Real-Time Dynamic Adjustments

The routing engine reacts to runtime signals to optimize selection:

• Rate-limit headers from providers cause immediate deprioritization of overloaded targets.
• Cost metrics from src/lib/pricingSync.ts enable the cost-optimized strategy to pick the cheapest viable provider.
• Context-usage statistics (e.g., token consumption) feed the context-optimized strategy, preferring providers with available headroom.

These adjustments happen during the execution loop without requiring configuration changes.

Fallback and Resilience Mechanisms

If all primary targets fail, OmniRoute activates the global emergency fallback implemented in open‑sse/services/emergencyFallback.ts. This fallback can route to a self-hosted model or an inexpensive OpenAI-compatible endpoint, guaranteeing response availability during provider outages. This resilience layer ensures that routing failures never result in complete service unavailability.

Configuring Routing in OmniRoute

OmniRoute exposes routing configuration through both CLI tools and API endpoints.

Creating a Combo via CLI

Define weighted routing across multiple providers:

omniroute combo create \
  --name "fast‑and‑cheap" \
  --strategy weighted \
  --targets '[{"provider":"openai","model":"gpt-4o","weight":70},{"provider":"groq","model":"llama3-70b","weight":30}]'

This command persists the combo to the SQLite combos table and registers the weighted strategy.

Using Combos in API Requests

Reference combos using the combo: prefix in model parameters:

POST /api/v1/chat/completions
{
  "model": "combo:fast‑and‑cheap",
  "messages": [{ "role": "user", "content": "Explain quantum tunneling." }]
}

Inside open‑sse/services/combo.ts, the model string triggers resolveComboTargets() to fetch the configuration and handleComboChat() to execute the weighted selection logic.

Monitoring Routing Performance

Access execution metrics programmatically:

import { getComboMetrics } from "@omniroute/open-sse/mcp-tools";

const metrics = await getComboMetrics({ comboId: "fast‑and‑cheap" });
console.log(metrics); // { successRate: 0.97, avgLatencyMs: 210, costPerMTokens: 0.002 }

Metrics are aggregated from execution logs stored in src/lib/db/comboMetrics.ts and help operators tune strategies based on actual performance data.

Summary

• OmniRoute handles routing through the Combo Routing Engine, which processes requests via configurable provider sequences stored in src/lib/db/combo.ts.
• 17 routing strategies—including weighted, cost-optimized, and context-aware—determine target traversal order, with identifiers defined in src/shared/constants/routingStrategies.ts.
• The execution loop in open‑sse/services/combo.ts iterates through ResolvedComboTarget objects until success, with circuit-breaker logic and error handling.
• Dynamic adjustments respond to rate limits, pricing updates from src/lib/pricingSync.ts, and context usage in real-time.
• Emergency fallback via open‑sse/services/emergencyFallback.ts ensures availability when all primary targets fail.

Frequently Asked Questions

What is a combo in OmniRoute?

A combo is a routing configuration that defines one or more provider-model pairs with optional weights, priorities, and fallback rules. Combos are stored in the SQLite database via src/lib/db/combo.ts and serve as the foundational unit for OmniRoute's routing decisions, allowing administrators to treat multiple providers as a single logical endpoint.

How does OmniRoute choose which provider to use?

OmniRoute selects providers based on the routing strategy attached to the combo configuration. The resolveComboTargets() function in open‑sse/services/combo.ts expands the combo into ordered targets, and handleComboChat() traverses them according to strategies like weighted distribution, cost-optimization using data from src/lib/pricingSync.ts, or priority ordering defined in src/shared/constants/routingStrategies.ts.

What happens if all providers in a combo fail?

If all targets fail, OmniRoute triggers the global emergency fallback located in open‑sse/services/emergencyFallback.ts. This routes the request to a self-hosted model or inexpensive backup endpoint, ensuring continuous availability even during widespread provider outages.

How many routing strategies does OmniRoute support?

OmniRoute supports 17 distinct routing strategies, including priority-based, weighted, round-robin, P2C (Power of Two Choices), cost-optimized, and context-optimized approaches. The complete list of strategy identifiers is maintained in src/shared/constants/routingStrategies.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}