When to Use the Fill-First Combo Routing Strategy in OmniRoute

Use the fill-first combo routing strategy in OmniRoute when you need deterministic, priority-preserving failover that always attempts your primary provider first before falling back to secondary options.

The fill-first strategy serves as the default fallback mechanism for OmniRoute’s combo routing engine. It processes candidate provider connections in strict priority order, ensuring requests hit your preferred target until it becomes unavailable due to rate limits, cooldowns, or retry-eligible errors. This approach is ideal for maintaining clear hierarchies between primary and backup LLM providers while benefiting from automatic failover capabilities.

What Is the Fill-First Strategy?

The fill-first strategy operates as a sequential selection algorithm within OmniRoute’s combo routing system. It walks the list of candidate provider connections in fixed priority order and sends every request to the first viable target. Only when that target is unavailable—because it is rate-limited, has a cooldown, or encounters a retry-eligible error—does the engine “fill” the request onto the next target in the list.

Core Mechanism

In open-sse/services/combo/strategyDispatch.ts, the routing engine evaluates the combo’s connection array sequentially. The implementation in open-sse/services/combo/applyStrategyOrdering.ts ensures the original ordering is respected: the first entry is always tried before the second, the second before the third, and so on. As soon as the leading connection reports a retry-eligible failure (e.g., HTTP 429, 500, 502, 503, or 504), the request is re-queued on the next connection without load-balancing or probabilistic selection.

Key Characteristics

• Priority preservation – The original ordering of the combo’s connections (defined by the user or system) remains intact throughout the request lifecycle.
• Simple fail-over – When the primary target fails, the system immediately shifts to the next available candidate in the sequence.
• Deterministic behavior – Because the order never changes, the same request sequence will always hit the same provider unless that provider becomes temporarily unavailable, making debugging and cost-prediction straightforward.
• Default selection – According to the source code in src/sse/services/auth.ts, if no fallbackStrategy is provided, the system automatically falls back to "fill-first".

When to Use Fill-First in OmniRoute

This strategy excels in scenarios requiring strict provider hierarchy and predictable routing behavior. Consider implementing fill-first for the following use cases:

Primary and Secondary Provider Combos

Use fill-first when operating a primary/secondary provider combo, such as routing to a paid commercial LLM first and a free open-source model as backup. This guarantees all traffic uses the paid model until it cannot, reducing unnecessary costs while preserving quality. The deterministic nature ensures you maximize utilization of your preferred provider before consuming secondary resources.

Rate-Limited API Accounts

When working with a single API key that may hit rate limits, fill-first keeps trying the same key until the limit is reached, then automatically switches to the next key without manual intervention. This is particularly valuable for high-throughput applications where temporary throttling should trigger immediate failover to alternate accounts in the same combo.

Deterministic Testing Environments

Fill-first is ideal for integration testing scenarios requiring reproducible routing. The fixed order makes it easy to assert which provider should have handled a specific request, allowing deterministic validation of provider-specific behaviors and error handling paths in your test suite.

Simplified Operational Monitoring

Choose fill-first when you want a single “golden” provider to dominate traffic for easier capacity planning. Because metrics concentrate on the top target until failover occurs, you can simplify alerting thresholds and reduce the noise associated with distributed load-balancing strategies.

Implementation Details

The fill-first strategy is hardcoded as the default behavior across the OmniRoute codebase. The selection logic resides in src/sse/services/auth.ts, where the system checks for an explicit fallbackStrategy configuration. If none is provided, it defaults to "fill-first" to ensure consistent behavior out of the box.

The strategy’s behavior is validated by comprehensive test suites:

• combo-routing-engine.test.ts confirms priority order preservation.
• combo-strategy-fallbacks.test.ts verifies fail-over to subsequent targets when the first connection fails.
• combo-runtime-unit-concurrency.test.ts validates overflow to the next unit when the first connection reaches capacity limits.

All supported strategies are enumerated in src/shared/constants/routingStrategies.ts, where "fill-first" is defined alongside alternatives like round-robin and least-connections.

Code Examples

Configuring a New Combo

When creating a combo via the database layer, fill-first is the implicit default:

// src/lib/db/combo.ts – create a combo; "fill-first" is the default fallback
await insertCombo(db, "my-combo", "qtSd/grp/openai/gpt-4o", "fill-first");

The combo will first try the openai/gpt-4o connection; if it is rate-limited, the next connection in the combo list will be used.

Runtime API Configuration

Explicitly specify the strategy in your API requests to ensure predictable behavior:

// Example POST to /v1/chat/completions
{
  "model": "my-combo",
  "fallbackStrategy": "fill-first",   // optional – same as the default
  "messages": [{ "role": "user", "content": "Explain fill-first" }]
}

Programmatic Account Selection

For custom middleware or authentication layers, use the account selector with the fill-first strategy:

import { selectAccount } from "@/open-sse/services/accountSelector";

const account = selectAccount(accounts, "fill-first");
// Returns the first account that is not on cooldown.

Resolving Combo Targets

When pre-resolving routing targets programmatically:

import { resolveComboTargets } from "@/open-sse/services/combo/targetResolution";

const targets = await resolveComboTargets({
  comboName: "my-combo",
  strategy: "fill-first",
});

The returned list will be ordered by priority; the runtime will attempt each in turn until one succeeds.

Summary

• Fill-first is the default combo routing strategy in OmniRoute when no fallbackStrategy is explicitly configured in src/sse/services/auth.ts.
• The strategy preserves strict priority order, ensuring your primary provider receives all traffic until it becomes unavailable.
• Ideal use cases include primary/secondary provider hierarchies, rate-limited account management, deterministic testing, and simplified operational monitoring.
• Implementation spans open-sse/services/combo/strategyDispatch.ts and open-sse/services/combo/applyStrategyOrdering.ts, with comprehensive test coverage in the combo test suite.

Frequently Asked Questions

Is fill-first the default strategy in OmniRoute?

Yes. According to the source code in src/sse/services/auth.ts, the system automatically falls back to "fill-first" when no explicit fallbackStrategy is provided in the request or configuration. This ensures consistent, predictable routing behavior immediately after installation.

How does fill-first differ from round-robin strategies?

While fill-first maintains a static priority order and only advances to the next provider upon failure, round-robin strategies distribute requests sequentially across all healthy providers regardless of priority. Fill-first optimizes for hierarchy preservation and cost control, whereas round-robin optimizes for load distribution.

Can I use fill-first with more than two providers?

Yes. Fill-first supports any number of connections within a combo. The engine will traverse the entire ordered list from first to last, attempting each connection in sequence until it finds one that is not rate-limited, cooling down, or erroring. This makes it suitable for complex chains involving primary, secondary, and tertiary fallbacks.

How do I debug routing decisions when using fill-first?

Because fill-first is deterministic, you can trace routing decisions by examining the ordered connection list in your combo configuration and checking provider availability metrics. The strategy guarantees that request N will always route to the same provider given the same health state, allowing you to correlate specific requests with specific providers in your logs without probabilistic noise.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}