OmniRoute Combo Execution with Composite Tiers: How the Tiered Routing Engine Works

OmniRoute combo execution processes requests through a prioritized stack of composite tiers, where each tier contains provider-model candidates and health-gate predicates that determine routing, fallback, and retry behavior.

OmniRoute, an open-source LLM routing gateway by diegosouzapw/OmniRoute, handles complex inference requests using a composite tier architecture. This design allows a single API call to traverse multiple prioritized tiers—each with distinct providers, models, and execution policies—until a successful response is returned or all options are exhausted.

How Composite Tiers Structure OmniRoute Combo Execution

The foundation of OmniRoute combo execution lies in the tier stack, a dynamic sequence of composite tiers resolved at request time. According to the source code in open-sse/services/comboSetup.ts, the system builds this stack from either the user-specified combo definition or the default auto-combo configuration defined in open-sse/services/comboConfig.ts.

Each composite tier is a lightweight object containing:

• A list of provider-model candidates
• Health-gate predicates (rate limits, circuit breakers, quotas)
• A fallback policy governing retry behavior

The structure is validated against the Zod schema in src/shared/validation/schemas/combo.ts before entering the execution pipeline. This ensures that every tier conforms to the expected interface before the engine attempts resolution.

The Composite Tier Execution Pipeline

When a request hits the /v1/chat/completions endpoint, the combo engine activates a multi-stage pipeline that evaluates and executes tiers sequentially.

Tier Resolution and Visibility

Before execution begins, open-sse/services/comboVisibility.ts evaluates whether a tier is visible to the current caller. This check incorporates service-tier entitlements, feature flags, and quota availability. Invisible tiers are silently skipped rather than failing the request, allowing the engine to proceed only through authorized routing paths.

Predicate Evaluation and Health Gates

For each visible tier, open-sse/services/comboPredicates.ts runs a battery of health-gate predicates. These predicates return one of three states: allow, retry-later, or skip. The checks include:

• Circuit breaker status
• Rate limit consumption
• Model lockout conditions
• Custom quota thresholds

Tiers failing these predicates are temporarily bypassed, and their cooldown status is tracked in comboCooldownRetry.ts.

The Core Execution Loop

The primary execution logic resides in open-sse/services/combo.ts. The engine iterates through the tier stack using the strategy defined in open-sse/services/combo/comboStructure.ts:

1. Select the next eligible model within the current tier
2. Dispatch the request via the appropriate executor (e.g., open-sse/executors/openai.ts)
3. On success, optionally hand off to a judge model (for fusion strategies) or return the response directly
4. On failure, capture the error context and evaluate fallback conditions

Fallback Handling and Error Aggregation

When a tier exhausts all candidates or encounters a non-retryable error, control passes to the next composite tier in the stack. The open-sse/services/comboErrorAggregation.ts module aggregates diagnostic information from failed attempts, while comboCooldownRetry.ts manages backoff timers to prevent rapid re-tries against unhealthy providers.

If all tiers are exhausted, open-sse/services/comboAbortReasons.ts generates a structured error outlining the specific failure modes (e.g., all tiers exhausted, provider circuit open, quota exceeded) and returns it to the client.

Runtime Configuration and Dynamic Tier Stacks

OmniRoute combo execution supports dynamic tier injection without service restarts. The /api/settings/tier-config endpoint allows operators to POST new tier definitions at runtime, immediately influencing subsequent request routing.

Service-tier overrides can reorder or inject tiers based on customer entitlements. Additionally, the health-gate logic can dynamically hide tiers when their underlying providers enter a circuit-breaker OPEN state, ensuring the stack remains responsive to real-time infrastructure conditions.

Code Examples

Executing a Combo via CLI

Use the --tier flag to explicitly define the composite tier order:

omniroute combo \
  --model "gpt-4o-mini" \
  --prompt "Explain quantum tunnelling in one sentence." \
  --tier priority,flex

If omitted, tiers resolve from the default auto-combo configuration.

Programmatic Node SDK Usage

import { createComboRequest } from '@omniroute/sdk';

const request = createComboRequest({
  model: 'gpt-4o-mini',
  prompt: 'Summarize the latest news about AI.',
  tiers: ['priority', 'flex', 'standard'], // composite tier order
});

const response = await fetch('https://localhost:20128/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(request),
});

const data = await response.json();
console.log(data.choices[0].message.content);

Injecting a Custom Tier via Settings API

curl -X POST https://localhost:20128/api/settings/tier-config \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
        "providerId": "openai",
        "tier": "premium",
        "modelList": ["gpt-4o", "gpt-4o-mini"]
      }'

This adds a premium tier for OpenAI, causing subsequent combo executions to prioritize these models before falling back to lower tiers.

Key Source Files in OmniRoute Combo Execution

The following files implement the composite tier mechanism in the diegosouzapw/OmniRoute repository:

• open-sse/services/combo.ts – Core execution loop, tier fallback, and error aggregation
• open-sse/services/comboSetup.ts – Tier stack construction from request or auto-combo config
• open-sse/services/comboPredicates.ts – Health-gate predicate evaluation (circuit breaker, quota, cooldown)
• open-sse/services/comboVisibility.ts – Tier visibility filtering by service tier and feature flags
• open-sse/services/comboErrorAggregation.ts – Error collection and formatting from failed tiers
• open-sse/services/comboCooldownRetry.ts – Cooldown management and retry timing logic
• open-sse/services/comboAbortReasons.ts – Enumeration of final failure reasons
• open-sse/services/combo/comboStructure.ts – Tier structure definitions and selection strategies
• src/shared/validation/schemas/combo.ts – Zod schema for combo payload validation
• bin/cli/commands/combo.mjs – CLI entry point for manual combo execution

Summary

• OmniRoute combo execution routes requests through a prioritized stack of composite tiers, where each tier contains multiple provider-model candidates.
• The pipeline validates requests against src/shared/validation/schemas/combo.ts, then resolves tiers via comboSetup.ts and filters them through comboVisibility.ts.
• Health predicates in comboPredicates.ts determine tier eligibility, while combo.ts manages the execution loop and fallback sequencing.
• Failed attempts are tracked in comboErrorAggregation.ts and comboCooldownRetry.ts, with final abort reasons enumerated in comboAbortReasons.ts.
• Runtime tier injection is supported via the /api/settings/tier-config endpoint, enabling dynamic routing strategies without service restarts.

Frequently Asked Questions

What is a composite tier in OmniRoute?

A composite tier is a logical routing layer containing a list of provider-model candidates, health-gate predicates, and fallback policies. According to the source code in open-sse/services/combo/comboStructure.ts, tiers act as sequential stages in the combo execution pipeline, allowing requests to cascade from high-priority providers to backup options when failures occur.

How does OmniRoute decide which tier to use first?

The tier order is determined by comboSetup.ts, which builds the stack from either the tiers array in the request payload or the default auto-combo configuration. Visibility filters in comboVisibility.ts then remove ineligible tiers based on service-tier entitlements and feature flags before execution begins.

What happens when all tiers fail during combo execution?

If the engine exhausts all composite tiers without success, comboAbortReasons.ts generates a structured error response detailing the specific failure modes—such as circuit-breaker states, quota exhaustion, or provider errors—and returns it to the client with a non-200 HTTP status.

Can I add custom tiers without restarting OmniRoute?

Yes. The /api/settings/tier-config endpoint accepts POST requests to inject or modify tiers at runtime. As implemented in open-sse/services/comboConfig.ts, these changes take effect immediately for subsequent requests, enabling zero-downtime routing adjustments and per-customer service-tier customizations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}