Does OmniRoute Support Dynamic Routing? A Deep Dive into the Combo Engine

Yes, OmniRoute supports fully dynamic routing through its Combo engine, which selects the best upstream model for every request at runtime using live telemetry and configurable strategies.

OmniRoute is an open-source AI model router that implements sophisticated dynamic routing capabilities. At the heart of this system lies the Combo engine, which re-evaluates routing decisions for every single request based on current provider health, latency statistics, and cost optimization policies. This article explores how OmniRoute achieves true dynamic routing without requiring server restarts or code deployments.

How Dynamic Routing Works in OmniRoute

Runtime Combo Resolution

The dynamic routing process begins in open-sse/services/combo.ts, where the resolveComboTargets function processes incoming requests. Unlike static routing configurations, OmniRoute stores combo definitions in SQLite via src/lib/db/combos.ts, allowing administrators to create, update, or delete routing rules without restarting the server.

When a request arrives containing a combo identifier (e.g., "model": "combo:dynamic-openai-anthropic"), the engine calls expandProviderWildcardsInCombo to resolve wildcard patterns like openai/* or anthropic/claude-* against the live model catalog. This expansion happens at request time, meaning new providers or models automatically become available as soon as they are registered.

Live Telemetry Integration

The buildAutoCandidates function (located around line 93 in open-sse/services/combo.ts) gathers real-time metrics to inform routing decisions. For each candidate model, OmniRoute queries:

• Latency statistics via getModelLatencyStats
• Pricing data via getPricingForModel
• Circuit-breaker state via getCircuitBreaker
• Quota usage and session stickiness requirements

This telemetry-driven approach ensures that unhealthy providers or exhausted quotas are bypassed automatically.

Strategy Evaluation

OmniRoute supports multiple routing strategies defined in src/shared/constants/routingStrategies.ts, including priority, weighted distribution, round-robin, and auto-combo. The applyStrategyOrdering function evaluates these strategies on-the-fly, while handleAutoCandidates uses the live telemetry data to score and rank available providers.

For sessions requiring stickiness, the system validates pinned models through isPinnedModelDurablyUnhealthy (around line 1000 in open-sse/services/combo.ts), dynamically dropping stale or unhealthy pinned entries in favor of viable alternatives.

Key Mechanisms Behind Dynamic Routing

• Combo Resolution: The resolveComboTargets function in open-sse/services/combo/comboStructure.ts transforms combo definitions into concrete target lists at request time.

• Wildcard Expansion: expandProviderWildcardsInCombo in open-sse/services/combo/providerWildcard.ts maps patterns like openai/* to current model catalog entries.

• Auto-Combo Pipeline: handleAutoCandidates and buildAutoCandidates evaluate latency, cost, and quota constraints to select optimal targets.

• Dynamic Updates: The combos database module in src/lib/db/combos.ts enables runtime modifications to routing configurations.

Implementing Dynamic Routing: Code Examples

Creating a Dynamic Combo with Wildcards

Define a combo that automatically includes all current and future OpenAI and Anthropic models:

import { createCombo } from "@/src/lib/db/combos";

// Define a combo that always uses the latest OpenAI models plus any
// Anthropic models that match the pattern `claude-*`.
await createCombo({
  name: "dynamic-openai-anthropic",
  models: [
    "openai/*",               // expands to every active OpenAI model
    "anthropic/claude-*"      // expands to all Claude models
  ],
  strategy: "auto",          // OmniRoute will pick the best target per request
  config: {
    maxRetries: 2,
    retryDelayMs: 1000,
  },
});

When a request arrives, the engine calls expandProviderWildcardsInCombo to turn these wildcards into the current list of available models.

Triggering Dynamic Routing via API

Send a request that triggers the Combo engine to evaluate and select the best provider:

curl -X POST https://router.example.com/v1/chat/completions \
  -H "Authorization: Bearer $OMNIROUTE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "combo:dynamic-openai-anthropic",
        "messages": [{"role":"user","content":"Explain quantum tunneling"}],
        "max_tokens": 512
      }'

The route handler in src/app/api/v1/chat/completions/route.ts processes this request by looking up the combo, expanding wildcards, evaluating live metrics, and routing to the optimal upstream provider.

Updating Routes at Runtime

Modify existing combos without restarting the server:

import { updateCombo } from "@/src/lib/db/combos";

await updateCombo("dynamic-openai-anthropic", {
  // Add a new provider wildcard
  models: ["openai/*", "anthropic/claude-*", "groq/*"],
});

The next incoming request instantly sees the new Groq targets because combo resolution happens per request.

Summary

• OmniRoute implements dynamic routing through its Combo engine, which re-evaluates routing decisions for every request.
• Wildcard patterns like openai/* expand at runtime to include current providers, eliminating the need for code changes when adding models.
• Live telemetry including latency, quota usage, and circuit-breaker states drives the auto-combo selection strategy.
• Routing configurations are stored in SQLite and can be updated at runtime without server restarts.
• The system supports multiple routing strategies (priority, weighted, round-robin, auto) evaluated on-the-fly via applyStrategyOrdering.

Frequently Asked Questions

Does OmniRoute require a server restart to update routing rules?

No. OmniRoute stores combo definitions in SQLite via src/lib/db/combos.ts. You can create, update, or delete combos using the database module, and changes take effect immediately on the next request. The routing engine reads the current configuration at request time, enabling true zero-downtime updates.

What routing strategies does OmniRoute support?

OmniRoute supports multiple strategies defined in src/shared/constants/routingStrategies.ts, including priority-based routing, weighted distribution, round-robin, and the auto-combo strategy. The auto strategy uses buildAutoCandidates to evaluate live telemetry such as latency statistics, pricing, and quota availability to select the optimal provider for each request.

How does OmniRoute handle provider failures in real-time?

The Combo engine integrates circuit-breaker patterns and health checks through functions like getCircuitBreaker and isPinnedModelDurablyUnhealthy. When a provider fails or becomes unhealthy, the system automatically excludes it from the candidate pool for subsequent requests. Session stickiness is also dynamically validated, allowing the router to drop pinned models that are durably unhealthy in favor of viable alternatives.

Can I use wildcards to automatically include new models?

Yes. The expandProviderWildcardsInCombo function in open-sse/services/combo/providerWildcard.ts resolves patterns like openai/* or anthropic/claude-* against the live model catalog at request time. When new providers register or existing providers add models, they automatically become eligible for routing without requiring configuration updates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}