Cache-Optimized vs Context-Optimized Routing in OmniRoute: A Complete Comparison

Cache-optimized routing pins requests to the same provider/account to maximize prompt-cache reuse, while context-optimized routing prioritizes models with the largest available context windows.

OmniRoute, the open-source AI routing engine by diegosouzapw/OmniRoute, provides multiple routing strategies that determine how a combo selects its target provider and model. Two specialized strategies—cache-optimized and context-optimized—address distinct performance needs: cache locality versus context capacity. Both strategies are defined in [src/shared/constants/routingStrategies.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/shared/constants/routingStrategies.ts#L95-L102) and implemented in the combo dispatcher via applyStrategyOrdering.

What Is Cache-Optimized Routing?

Cache-optimized routing maximizes prompt-cache reuse by keeping consecutive requests on the same provider/account. This reduces latency when the same prompt prefix is used repeatedly.

How Cache-Optimized Routing Works

The implementation in [open-sse/services/combo/promptCacheAffinity.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/combo/promptCacheAffinity.ts) follows four steps:

1. Resolve a cache affinity key from the request body—either an explicit prompt_cache_key or a deterministic prefix hash—via resolvePromptCacheAffinityKey (lines 94-115)

2. Expand the target list with concrete active connections using expandPromptCacheAffinityTargets to enable affinity targeting at the account level (lines 164-176)

3. Apply rendezvous hashing via applyPromptCacheAffinity, scoring targets with the resolved key; the highest-scoring target becomes first in the ordered list (lines 73-100)

4. Return the ordered list to the combo executor

The ordering is stable for a given cache key—repeated calls with identical keys hit the same account. The combo logger emits: Cache‑optimized ordering: … first (source).

When to Use Cache-Optimized Routing

• Chatbots with consistent prefixes that reuse the same system prompt
• High-frequency similar requests where provider-side prompt embeddings provide measurable latency reduction
• Workloads where cache locality outweighs cost or context considerations

Cache-optimized routing is mutually exclusive with most other ordering strategies. It runs after primary ordering (e.g., cost-optimized) and overrides the result based on cache affinity.

What Is Context-Optimized Routing?

Context-optimized routing prioritizes the largest available context window, ensuring the model can process the maximum amount of conversation history or document content.

How Context-Optimized Routing Works

The implementation centers on [open-sse/services/combo/comboStructure.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/combo/comboStructure.ts):

1. Retrieve each target's known context limit via getModelContextLimitForModelString

2. Sort all targets by context window size in descending order using sortTargetsByContextSize (lines 13-31)

3. Return the reordered list with the largest-context model first

The logger emits: Context‑optimized ordering: largest first … (source).

When to Use Context-Optimized Routing

• Long-form document analysis requiring maximum token capacity
• Multi-turn conversations with extensive history
• Scenarios where context capacity matters more than cache locality or cost

Context-optimized routing is a stand-alone ordering strategy that directly replaces any previous ordering. It does not combine with modifiers like cost-optimized because it completely reorders the target list.

Key Differences: Cache-Optimized vs Context-Optimized

Aspect	Cache-Optimized	Context-Optimized
Primary goal	Maximize prompt-cache reuse on the same provider/account	Maximize available context window size
Ordering mechanism	Rendezvous hashing with stable affinity key	Descending sort by context window size
Effect on target list	Pins first target; preserves subsequent order	Completely reorders entire list
Interaction with other strategies	Runs after and overrides other orderings	Replaces all other ordering logic
Best for	Repeated similar prompts, cache-sensitive workloads	Long documents, multi-turn conversations
Core source file	open-sse/services/combo/promptCacheAffinity.ts	open-sse/services/combo/comboStructure.ts

Code Examples: Defining Each Strategy

// Cache-optimized: keeps requests on same provider for cache warmth
const comboCacheOptimized = {
  name: "chat-cache-opt",
  strategy: "cache-optimized",
  targets: [
    { provider: "openrouter", modelStr: "anthropic/claude-3.5-sonnet" },
    { provider: "openrouter", modelStr: "openai/gpt-4o-mini" },
  ],
};

// Context-optimized: prioritizes largest context window
const comboContextOptimized = {
  name: "doc-review-context-opt",
  strategy: "context-optimized",
  targets: [
    { provider: "openrouter", modelStr: "google/gemini-2.5-flash" },  // 1,048,576 tokens
    { provider: "openrouter", modelStr: "openai/gpt-4o-mini" },       // 128,000 tokens
  ],
};

// Both are invoked through the same handler
await handleComboChat(buildRequest({ body: { model: "chat-cache-opt", ... } }));
await handleComboChat(buildRequest({ body: { model: "doc-review-context-opt", ... } }));

Source Code Reference

Understanding the implementation requires familiarity with these key files in diegosouzapw/OmniRoute:

File	Purpose
[src/shared/constants/routingStrategies.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/shared/constants/routingStrategies.ts)	Enum defining cache-optimized and context-optimized as valid strategy values
[open-sse/services/combo/applyStrategyOrdering.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/combo/applyStrategyOrdering.ts)	Central dispatch implementing both strategy branches
[open-sse/services/combo/promptCacheAffinity.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/combo/promptCacheAffinity.ts)	Cache affinity resolution, expansion, and rendezvous hashing
[open-sse/services/combo/comboStructure.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/combo/comboStructure.ts)	sortTargetsByContextSize for context-based ordering

Summary

• Cache-optimized routing leverages resolvePromptCacheAffinityKey, expandPromptCacheAffinityTargets, and applyPromptCacheAffinity to pin requests to consistent provider accounts, reducing latency through prompt-cache reuse.

• Context-optimized routing uses sortTargetsByContextSize in comboStructure.ts to reorder targets purely by context window capacity, ensuring maximum available memory for each request.

• The strategies are fundamentally incompatible with each other—cache-optimized modifies existing ordering while context-optimized replaces it entirely.

• Choose cache-optimized for repetitive, cache-sensitive workloads and context-optimized for memory-intensive, long-context applications.

Frequently Asked Questions

Can I combine cache-optimized and context-optimized routing in the same combo?

No. According to the OmniRoute source code, these strategies operate differently—cache-optimized runs after and overrides other orderings, while context-optimized completely replaces the target list order. The applyStrategyOrdering dispatcher processes only one strategy per combo request. You must select the strategy that better aligns with your performance priority.

How does cache-optimized routing handle multiple active connections to the same provider?

The expandPromptCacheAffinityTargets function (lines 164-176) expands the abstract target list into concrete active connections, enabling affinity at the account level. Rendezvous hashing then deterministically selects one specific connection based on the resolved affinity key, ensuring stable routing even with multiple endpoints.

Does context-optimized routing consider cost or latency?

No. The sortTargetsByContextSize implementation (lines 13-31) sorts purely by context window size in descending order. It does not factor in pricing, latency, or other performance metrics. For cost-conscious context-heavy workloads, you may need to manually order targets or use a different strategy.

What happens if the cache-optimized target becomes unavailable?

The cache-optimized strategy produces an ordered list, not a single target. If the affinity-pinned first target fails, the combo executor falls through to subsequent targets in the ordered list. However, fallback targets will not share the same prompt-cache state, potentially increasing latency for that specific request.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}