How OmniRoute's Combo Routing Engine Works Internally: A 12-Step Architecture Breakdown

OmniRoute's combo routing engine selects optimal provider-model pairs through a 12-stage pipeline that creates execution contexts, resolves configuration, handles pinned models, expands wildcards, applies strategy-specific ordering, enforces session stickiness, and executes targets with intelligent failover and quality validation.

The combo routing engine in the diegosouzapw/OmniRoute repository serves as the core orchestration layer for intelligent LLM load distribution. Implemented primarily in open-sse/services/combo.ts, this system processes combo definitions—logical collections of models and routing strategies—to determine the best execution path for every incoming request.

Step 1: Context Initialization and Configuration Resolution

The engine begins execution by wrapping the request body, combo definition, settings, and logger into a unified ComboContext via createComboContext() in open-sse/services/combo.ts. This context object prepares all data for the subsequent pipeline stages.

Immediately following context creation, phaseComboSetup() extracts the routing strategy, config object, resilienceSettings, pinnedModel status, and timeout values from the combo definition. These parameters govern all downstream routing decisions.

Step 2: Pinned Model Handling and Strategy Shortcuts

Before expanding target lists, the engine checks for context cache pins from previous conversation turns. If a model was previously pinned, the engine attempts direct routing after validating through isPinnedModelDurablyUnhealthy that the provider remains healthy and the pin is still present in the combo definition.

For fusion and pipeline strategies, the engine takes early exits from standard resolution logic. These complex multi-model strategies delegate immediately to handleFusionChat and handlePipelineChat in open-sse/services/combo/fusion.ts and open-sse/services/combo/pipeline.ts respectively, as they require specialized control flows distinct from single-model routing.

Step 3: Wildcard Expansion and Target Resolution

The resolveComboTargets() function (or its weighted variant) transforms abstract combo definitions into concrete, ordered lists of ResolvedComboTarget objects. This critical phase performs several validation steps:

• Wildcard Expansion: Provider wildcards like openai/* are resolved into specific model entries via expandProviderWildcardsInCombo
• Health Checking: Targets are filtered through isProviderInCooldown and isModelLocked predicates to exclude unhealthy providers
• Auto-Combo Generation: When using the auto strategy, buildAutoCandidates generates eligible provider candidates scored by cost, latency, and quota availability
• Session Binding: applySessionStickiness prioritizes models recently used in the current session to improve cache hit rates

Step 4: Strategy-Specific Ordering Logic

Based on the strategy parameter extracted during setup, the engine applies distinct ordering algorithms:

Simple Ordering (Priority, Round-Robin, Random, Strict-Random, Fill-First): These strategies use applyStrategyOrdering in open-sse/services/combo/applyStrategyOrdering.ts to arrange targets in deterministic or randomized sequences.

Weighted Sticky Routing: This strategy maintains state in weightedStickyTargets (defined in open-sse/services/combo/rrState.ts), using execution keys to bias future selections toward previously successful targets for a configurable number of calls.

Auto Strategy: Dynamically scores candidates via scoreAutoTargets in open-sse/services/combo/autoStrategy.ts, weighing factors including cost per million tokens, latency percentiles, quota headroom, and reset-window affinity.

Step 5: Session Stickiness and Task-Aware Reordering

After primary strategy ordering, the engine applies two additive reordering layers that never override the base strategy:

1. Session Stickiness: Additional calls to applySessionStickiness reinforce recent model preferences for conversational continuity
2. Task Detection: reorderByTaskWeight in open-sse/services/combo/taskAwareRouting.ts analyzes the request to detect tasks (e.g., code completion versus summarization) and reorders targets based on learned performance weights for specific task types

Step 6: Target Execution and Timeout Handling

The engine executes targets through handleSingleModelWithTimeout, which wraps the user-provided handleSingleModel callback with per-target timeout logic via buildTargetTimeoutRunner.

Simple strategies invoke executeRuntimeUnitCombo() while round-robin variants use handleRoundRobinCombo from open-sse/services/combo/comboStructure.ts. Each target receives the original request body and a ResolvedComboTarget object containing provider, modelStr, connectionId, and executionKey.

Step 7: Failure Handling and Fallback Logic

When targets return recoverable errors—specifically HTTP 429 (rate limit), 500 (server error), or 400 (context overflow)—the engine records the failure via recordProviderFailure or recordModelLockoutFailure and automatically proceeds to the next target in the ordered list.

Non-retryable errors return immediately to the caller. The failure recording mechanism in open-sse/services/combo/comboCooldownRetry.ts enables exponential backoff and temporary provider exclusion.

Step 8: Quality Validation and Observability

Successful responses undergo validateResponseQuality checks against the combo's responseValidation rules. If validation fails (e.g., response format mismatch or content policy violations), the engine treats the result as a soft failure and falls back to subsequent targets.

Every routing attempt is recorded via recordComboRequest and recordComboShadowRequest for analytics. The engine emits detailed events through emit and notifyWebhookEvent to enable real-time monitoring and webhook integrations.

Core Data Structures and Type Definitions

The engine relies on strict typing defined in open-sse/services/combo/types.ts:

• ResolvedComboTarget: Normalized object containing provider, modelStr, connectionId, and executionKey
• ComboContext: Encapsulates request body, combo definition, settings, and logger instances
• weightedStickyTargets: Map structure in open-sse/services/combo/rrState.ts maintaining sticky state for weighted selections

Practical Implementation Examples

Invoking a Combo from an API Route

import { handleComboChat } from '@/open-sse/services/combo';
import { getComboFromData } from '@/open-sse/services/combo/comboStructure';

// Inside a Next.js API route handler
export async function POST(req: Request) {
  const body = await req.json();
  const combo = await getComboFromData('my-combo-name'); // fetch combo definition from DB
  return handleComboChat({
    body,
    combo,
    handleSingleModel: async (b, model) => {
      // use the default executor to call the upstream provider
      const exec = await getExecutor(model.provider);
      return exec.execute(b, model);
    },
    log: console,
    settings: {},          // optional global settings
    allCombos: null,       // not needed unless combo refs are used
  });
}

Recording Weighted Sticky Target Success

import { recordStickyWeightedSuccess } from '@/open-sse/services/combo/rrState';

// After a successful weighted combo execution
recordStickyWeightedSuccess(
  'my-combo-name',           // combo name
  execution.unit.executionKey, // execution key of the target that succeeded
  5                          // sticky limit (e.g., keep this target for the next 5 calls)
);

Customizing Auto-Combo Scoring Logic

import { buildAutoCandidates, scoreAutoTargets } from '@/open-sse/services/combo/autoStrategy';

// Custom scoring function that prefers lower cost over latency
function myScoring(candidates) {
  return candidates
    .map(c => ({ ...c, score: 1 / c.costPer1MTokens }))
    .sort((a, b) => b.score - a.score);
}

// Use within the combo handling flow
const candidates = await buildAutoCandidates(targets, combo.name);
const ranked = myScoring(candidates);

Summary

• OmniRoute's combo routing engine operates through a 12-step pipeline beginning with createComboContext() in open-sse/services/combo.ts and ending with quality validation and metrics emission.
• Strategy diversity includes simple ordering (priority, round-robin), weighted sticky selection using weightedStickyTargets, auto-scoring via buildAutoCandidates, and specialized fusion/pipeline handlers.
• Resilience mechanisms include pinned model validation, provider cooldown checks (isProviderInCooldown), automatic failover with recordProviderFailure, and response quality validation.
• Performance optimizations implement session stickiness through applySessionStickiness and task-aware routing via reorderByTaskWeight without overriding primary strategy decisions.
• Wildcard support allows dynamic model selection using patterns like openai/*, expanded through expandProviderWildcardsInCombo before target resolution.

Frequently Asked Questions

What is the entry point for combo routing in OmniRoute?

The primary entry point is handleComboChat exported from open-sse/services/combo.ts. This function accepts a ComboContext containing the request body, combo definition, and a handleSingleModel callback, then orchestrates the entire 12-step routing pipeline to select and execute the appropriate provider-model pair.

How does the weighted strategy maintain stickiness across requests?

The weighted strategy persists selection state in the weightedStickyTargets map defined in open-sse/services/combo/rrState.ts. When a target succeeds, recordStickyWeightedSuccess stores its execution key with a sticky limit count. Subsequent requests for the same combo check this map first, biasing selection toward recently successful targets until the sticky limit decrements to zero.

What happens when a provider returns a 429 or 500 error during combo execution?

The engine categorizes HTTP 429 (rate limit), 500 (server errors), and specific 400 errors (context overflow) as recoverable failures. It invokes recordProviderFailure to update cooldown state, then automatically attempts the next target in the resolved list. Non-recoverable errors return immediately without retrying additional targets.

How does the auto-combo strategy select candidates without explicit configuration?

The auto strategy generates candidates dynamically through buildAutoCandidates in open-sse/services/combo/autoStrategy.ts, then scores them using scoreAutoTargets. The scoring algorithm weighs cost per million tokens, latency percentiles, available quota, and reset-window affinity to rank providers automatically, requiring no predefined model list in the combo configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}