OmniRoute Combo Routing Strategies: A Complete Guide to 19 Load-Balancing Methods

OmniRoute supports 19 distinct combo routing strategies ranging from simple round-robin and random selection to advanced multi-factor optimization, including priority-based failover, weighted distribution, power-of-two-choices, and sophisticated approaches like fusion and context-relay.

OmniRoute's request pipeline distributes incoming requests across multiple AI providers using configurable combo routing strategies. These strategies, implemented in open-sse/services/combo.ts, determine how the platform selects target providers for load balancing, cost optimization, and resilience. Understanding these built-in strategies allows developers to fine-tune request distribution based on latency, cost, quota availability, and model capabilities.

What Are Combo Routing Strategies?

Combo routing is OmniRoute's mechanism for distributing a single request across multiple provider connections. When a request enters the system, the combo routing logic selects one or more target providers according to a configurable strategy.

The core logic lives in open-sse/services/combo.ts, which resolves the chosen strategy name to a concrete handler function. Each strategy implements a different load-balancing or fallback policy, giving operators granular control over traffic distribution.

The 19 Built-In Combo Routing Strategies

OmniRoute ships with 19 public strategies, each optimized for specific operational requirements:

Basic Distribution Strategies

These strategies provide straightforward load balancing without complex heuristics:

• priority – Sends traffic to the highest-priority provider first, falling back to the next priority on failure.
• weighted – Distributes requests proportionally to user-defined weights per provider.
• fill-first – Fills a provider's quota completely before moving to the next one.
• round-robin – Cycles through providers in a fixed order, giving each an equal share.
• random – Chooses a provider uniformly at random.
• strict-random – Random selection that respects strict policy constraints such as model compatibility.

Performance and Load-Based Strategies

These strategies optimize for system performance and current load conditions:

• p2c (Power-of-Two-Choices) – Randomly picks two providers and selects the one with lower load, balancing traffic more effectively than pure random selection.
• least-used – Selects the provider that has handled the fewest recent requests.
• headroom – Prefers providers with the most remaining quota (headroom) available.

Cost and Optimization Strategies

These strategies focus on economic efficiency and intelligent selection:

• cost-optimized – Prefers the cheapest provider that satisfies the request constraints.
• auto – The Auto-Combo engine automatically scores providers across 15 factors and chooses the best overall. See docs/routing/AUTO-COMBO.md for implementation details.
• lkgp (Least-Known-Good-Provider) – Chooses the provider with the longest history of successful calls, prioritizing reliability.

Resilience and Timing Strategies

These strategies handle circuit breakers and reset windows:

• reset-aware – Takes into account provider-level circuit-breaker reset windows when selecting a target.
• reset-window – Similar to reset-aware but focuses specifically on the next reset time window.

Context and Capability Strategies

These strategies route based on request characteristics and caching:

• context-optimized – Routes based on the amount of context the provider can handle.
• cache-optimized – Prefers providers that have cached results for the request.
• context-relay – Relays the request's context to multiple providers in parallel, merging results.

Advanced Multi-Provider Strategies

These strategies enable sophisticated request processing across multiple providers:

• fusion – Fans out to a panel of providers, then a special judge model synthesizes a single final answer. The implementation resides in open-sse/services/fusion.ts.
• pipeline – Chains providers so the output of one becomes the input of the next, enabling multi-step processing workflows.

Technical Implementation of Combo Routing

The combo routing system processes requests through a centralized dispatcher. In open-sse/services/combo.ts, the resolveComboTargets function parses the strategy field from incoming requests and invokes the appropriate handler.

The API layer integrates this through src/app/api/v1/chat/completions/route.ts, which forwards requests to handleChatCore in open-sse/handlers/chatCore.ts. This handler coordinates combo resolution with caching, rate-limiting, and response translation.

Implementing Combo Routing in Practice

Invoking the Auto Strategy

The auto strategy leverages OmniRoute's 15-factor scoring engine:

import { handleChatCore } from '@/open-sse/handlers/chatCore';
import { resolveComboTargets } from '@/open-sse/services/combo';

async function runAutoCombo(requestBody: any) {
  const combos = await resolveComboTargets({ 
    strategy: 'auto', 
    models: ['gpt-4o', 'claude-3'] 
  });
  return await handleChatCore(requestBody, combos);
}

Using Fusion for Consensus Answers

The fusion strategy aggregates responses from multiple providers through a judge model:

import { resolveComboTargets } from '@/open-sse/services/combo';

async function runFusionCombo(messages: any[]) {
  const targets = await resolveComboTargets({
    strategy: 'fusion',
    judgeModel: 'gpt-4o-mini',
    participants: ['gpt-4o', 'claude-3', 'gemini-1.5']
  });
  return targets[0];
}

Weighted Distribution via REST API

Configure proportional routing directly through the chat completions endpoint:

curl -X POST https://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "combo:weighted",
        "weights": {"openai:gpt-4o":0.7, "anthropic:claude-3":0.3},
        "messages":[{"role":"user","content":"Explain combo routing"}]
      }'

Summary

• OmniRoute provides 19 built-in combo routing strategies ranging from simple random selection to sophisticated multi-factor optimization.
• The core routing logic resides in open-sse/services/combo.ts, with specialized implementations like fusion in open-sse/services/fusion.ts.
• Strategies cover use cases including load balancing (round-robin, p2c), cost optimization (cost-optimized, auto), resilience (priority, reset-aware), and advanced processing (fusion, pipeline).
• The auto strategy evaluates 15 different factors to dynamically select optimal providers.
• All strategies integrate with the central resolveComboTargets function and work seamlessly with OmniRoute's caching and rate-limiting infrastructure.

Frequently Asked Questions

How do I choose the right combo routing strategy for my workload?

Select priority or weighted for predictable traffic distribution when you understand your provider performance characteristics. Use auto when you want OmniRoute to dynamically optimize across 15 factors including latency, cost, and reliability. For high-stakes applications requiring consensus, implement fusion to aggregate multiple provider responses through a judge model.

What is the difference between the fusion and pipeline strategies?

Fusion fans out requests to multiple providers simultaneously and uses a judge model to synthesize a single final answer, ideal for improving response quality through consensus. Pipeline chains providers sequentially, where the output of one provider feeds into the next, enabling multi-step processing workflows like refinement or translation chains.

Where is the combo routing logic implemented in the OmniRoute codebase?

The primary dispatcher lives in open-sse/services/combo.ts, which resolves strategy names to handler functions. Strategy-specific implementations like fusion reside in open-sse/services/fusion.ts. The API entry point at src/app/api/v1/chat/completions/route.ts forwards requests to handleChatCore in open-sse/handlers/chatCore.ts, which coordinates combo resolution with the rest of the request pipeline.

Can I use combo routing with custom weights and provider selections?

Yes. The weighted strategy accepts user-defined weights per provider through the API, as shown in the REST example. You can also specify exact model lists when invoking resolveComboTargets, allowing fine-grained control over which providers participate in each combo routing decision.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}