How OmniRoute Handles Dynamic Route Changes and Re‑routing: A Complete Guide

OmniRoute resolves routing decisions at runtime through its Combo Engine, which continuously evaluates provider health, quotas, and latency to automatically re‑route requests to alternative targets without client‑side intervention or server restarts.

OmniRoute is an open‑source AI model aggregation framework that treats routing as a dynamic, stateful decision. Instead of static configuration, the Combo Engine—implemented across open-sse/services/combo.ts and related modules—monitors upstream signals in real time and executes transparent failover logic.

The Combo Engine Architecture

Dynamic routing in OmniRoute centers on the concept of a Combo: a declarative list of target providers and models stored in the database. Each combo defines not only the candidate targets but also the credentials, weighting, and predicates that govern selection.

Combo definitions are managed in src/lib/db/combos.ts, where CRUD operations persist routing tables to the database. The structure of every combo is strictly validated by src/shared/validation/schemas/combo.ts, ensuring that runtime mutations adhere to type safety constraints. The runtime behavior is further controlled by constants defined in src/shared/constants/comboConfigMode.ts.

Runtime Routing Decisions and Strategies

When a request enters the chat handler, the Combo Engine invokes resolveComboTargets() from open-sse/services/combo/comboSetup.ts. This function expands the stored combo into an ordered array of ResolvedComboTarget objects based on the active routing strategy.

Strategies are defined in src/shared/constants/routingStrategies.ts and include:

• Priority – Targets are tried in strict order until one succeeds.
• Weighted – Traffic is distributed proportionally according to assigned weights.
• Round‑Robin – Requests cycle through targets sequentially.
• Least‑Used – Selects the target with the lowest current request count.
• Headroom – Routes to the provider with the most remaining quota capacity.

These strategies can be changed at runtime via the Combo Settings UI or the REST API, allowing operators to shift traffic patterns instantly.

Health‑Aware Fallback and Automatic Retry

Before a target is invoked, OmniRoute validates its viability through a health guard layer. This prevents wasted calls to degraded or rate‑limited providers.

Target Health Evaluation

The comboHealth.checkTargetHealth() function in src/lib/usage/comboHealth.ts evaluates recent success rates, quota consumption, and administrative cooldown periods. If a target is marked unhealthy, it is excluded from the candidate list for that request. The dashboard logic in src/lib/usage/comboHealthDashboard.ts provides visibility into these health states.

Cooldown and Retry Mechanisms

When a request fails against a chosen target, the engine triggers the retry logic located in open-sse/services/combo/comboCooldownRetry.ts. This module places the failed target into a temporary cooldown state—duration is configurable—and automatically retries the request against the next healthy candidate in the resolved list. The behavior is verified by the test suite in combo-cooldown-retry.test.ts.

Dynamic Re‑configuration Without Downtime

Administrators can modify combo composition, weights, or routing strategies through the REST API endpoints under src/app/api/settings/combo/ (for example, updateCombo.ts). Because resolveComboTargets() queries the database on every request, changes are reflected immediately without requiring a process restart.

// Example: Trigger a dynamic re‑route by updating a combo at runtime
import { db } from '@/lib/db';
await db.combos.updateCombo('default-chat', {
  targets: [
    { providerId: 'openai', model: 'gpt-4o', weight: 70 },
    { providerId: 'anthropic', model: 'claude-3-5-sonnet', weight: 30 },
  ],
  routingStrategy: 'weighted',
});

Streaming‑Aware Re‑routing for SSE Connections

For Server‑Sent Events (SSE) streams, failure detection must handle mid‑stream connection drops. The open-sse/services/combo/comboCompatFallback.ts module monitors upstream response status continuously. If the upstream aborts or returns an error status mid‑stream, the fallback logic terminates the current upstream request and transparently re‑invokes the Combo Engine with the next healthy target, preserving the client’s SSE connection without interruption.

// Example: Low‑level combo invocation that enables automatic re‑routing
import { handleComboChat } from '@/open-sse/services/combo';
const response = await handleComboChat(requestBody, {
  comboId: 'default-chat',
  // The engine automatically selects a healthy target and handles failover
});

Summary

• Combo Engine – The core runtime component that resolves provider‑model pairs on every request.
• Dynamic Strategies – Five routing strategies (priority, weighted, round‑robin, least‑used, headroom) configurable via src/shared/constants/routingStrategies.ts.
• Health Guard – Automatic exclusion of unhealthy targets via src/lib/usage/comboHealth.ts before invocation.
• Cooldown Retry – Failed targets enter a temporary ban period managed by open-sse/services/combo/comboCooldownRetry.ts.
• Zero‑Downtime Updates – Runtime changes to combos via src/app/api/settings/combo/ take effect immediately.
• Streaming Resilience – Mid‑stream failures in SSE connections trigger transparent re‑routing through open-sse/services/combo/comboCompatFallback.ts.

Frequently Asked Questions

How does OmniRoute decide when to re‑route a request?

OmniRoute evaluates three signals before and during each request. First, comboHealth.checkTargetHealth() in src/lib/usage/comboHealth.ts verifies that the target has not exceeded error thresholds or quota limits. Second, if an actual request fails, the cooldown logic in open-sse/services/combo/comboCooldownRetry.ts marks the target as unavailable for a configurable duration and selects the next candidate. Third, for streaming responses, comboCompatFallback.ts monitors for connection drops and triggers instant failover.

Can I change routing strategies while the server is running?

Yes. Routing strategies are stored in the database alongside combo definitions in src/lib/db/combos.ts. Because resolveComboTargets() reads the latest database state on every invocation, updating a combo’s strategy via the REST API or the Combo Settings UI immediately changes traffic distribution without requiring a server restart.

What happens if a provider fails mid‑stream during an SSE response?

The comboCompatFallback.ts service detects upstream disconnections or error status codes during active streams. It aborts the failing upstream request and re‑invokes the Combo Engine to select the next healthy target, then resumes streaming to the client from the new provider. This process is transparent to the caller and maintains the SSE connection state.

How does the cooldown mechanism prevent cascading failures?

When a target fails, open-sse/services/combo/comboCooldownRetry.ts applies a temporary cooldown period during which the target is automatically excluded from the candidate pool. This prevents the system from hammering an already degraded provider with retries, allowing upstream services time to recover while traffic shifts to healthy alternatives.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}