How OmniRoute Calculates Live Provider Pricing for Cost-Optimized Routing

OmniRoute determines the cheapest available provider at request time by syncing external price catalogs into an in-memory map, then ranking candidates by their input cost per million tokens while preserving original order on ties.

OmniRoute is an open-source AI model router (diegosouzapw/OmniRoute) that enables cost-optimized routing by calculating live provider pricing on every request. Rather than relying on static configuration, the system pulls fresh rates from the LiteLLM public catalog and applies them through a pipeline of sync services, in-memory lookups, and sorting algorithms. This ensures requests are always routed to the most cost-effective available model.

The Live Pricing Data Pipeline

OmniRoute maintains current pricing through a three-stage pipeline that connects external data sources to the request path.

Syncing from the LiteLLM Catalog

The system periodically fetches the public LiteLLM price database (model_prices_and_context_window.json) through the sync routine in src/lib/pricingSync.ts. This function updates the internal provider pricing table without overwriting user-defined overrides, ensuring that custom price settings persist across updates.

In-Memory Price Lookup

When a request enters the routing layer, the lookupPricing(model) function in src/mitm/inspector/pricing.ts queries an in-memory pricing map for the model's inputPerMTok and outputPerMTok values. This lookup occurs at request time to guarantee that the router uses the most recent rates available in memory.

Fallback Default Pricing

If lookupPricing cannot locate a specific model entry, the system falls back to hard-coded defaults defined in src/shared/utils/costEstimator.ts (lines 18–30). These defaults ensure the routing logic remains functional even when new models are added to the catalog before the sync routine executes.

Cost Estimation at Request Time

Before routing occurs, OmniRoute calculates expected costs using the shared estimateCost utility. This function converts input and output token counts into USD using the price-per-1M-tokens values retrieved during the lookup phase.

The same logic powers the preflightEstimate helper, which the dashboard uses to display real-time cost projections. Located in src/shared/utils/costEstimator.ts (lines 87–99), this utility ensures that the UI reflects the identical pricing data used by the proxy layer.

Cost-Optimized Routing Strategy

The cost-optimized combo-routing strategy implemented in open-sse/services/combo.ts uses live pricing data to sort candidate providers by economic efficiency rather than latency or quality scores.

Ranking by Input Price

The strategy calls getProviderPricing to retrieve current rates for each candidate model, then sorts the list by the cheapest input price (USD per 1M tokens). This prioritizes providers with lower prompt costs, which typically dominate total inference expenses.

Deterministic Tie-Breaking

When two providers offer identical input prices, OmniRoute preserves the original candidate order to ensure deterministic routing behavior. This logic is verified by the unit test cost-optimized preserves the original order on price ties in tests/unit/combo-strategy-fallbacks.test.ts (lines 243–250).

Reference Implementation Examples

The following examples demonstrate how to interact with OmniRoute's pricing utilities in application code.

Estimating Pre-Flight Costs

Use preflightEstimate to calculate expected costs before sending a request:

import { preflightEstimate } from '@/shared/utils/costEstimator';

const body = {
  system: "You are a helpful assistant.",
  messages: [{ role: "user", content: "Explain quantum tunneling." }],
  max_tokens: 500,
};

const model = "gpt-4o";
const { totalCost, formatted } = preflightEstimate(body, model);
console.log(`Estimated cost: $${formatted}`); // → $0.0012 (example)

Implementing Cost-Optimized Selection

To replicate the router's provider selection logic:

import { getProviderPricing } from '@/mitm/inspector/pricing';
import { sortByCostOptimized } from '@/open-sse/services/combo';

async function chooseProvider(models: string[]) {
  const priced = await Promise.all(models.map(async (m) => ({
    model: m,
    price: await getProviderPricing(m), // { inputPerMTok, outputPerMTok }
  })));
  const sorted = sortByCostOptimized(priced);
  return sorted[0].model; // cheapest input price
}

Summary

• Live data source: OmniRoute syncs the LiteLLM public catalog via src/lib/pricingSync.ts to maintain current rates.
• Fast lookup: The lookupPricing function in src/mitm/inspector/pricing.ts retrieves inputPerMTok and outputPerMTok from an in-memory map at request time.
• Default protection: Missing model entries fall back to hard-coded defaults in src/shared/utils/costEstimator.ts.
• Cost calculation: The estimateCost utility converts token counts to USD using live rates, accessible via preflightEstimate for UI previews.
• Routing logic: The cost-optimized strategy in open-sse/services/combo.ts sorts providers by input price, preserving original order on ties to ensure deterministic behavior.

Frequently Asked Questions

How often does OmniRoute update its pricing data?

The sync routine in src/lib/pricingSync.ts periodically fetches the LiteLLM model_prices_and_context_window.json catalog and updates the database without overwriting user-defined overrides. The in-memory lookup always uses the most recent sync results available, ensuring the router never uses stale rates.

What happens if a provider's price is not in the database?

If lookupPricing cannot find a model entry in the in-memory map, the system falls back to default pricing values defined in src/shared/utils/costEstimator.ts (lines 18–30). This ensures routing continues uninterrupted while administrators can update the pricing table independently.

Does the cost-optimized strategy consider output token prices?

The primary sort key is the input price per million tokens (inputPerMTok). While the getProviderPricing function retrieves both inputPerMTok and outputPerMTok, the cost-optimized ranking prioritizes input costs, as prompt tokens typically constitute the majority of variable inference expenses in most workloads.

How can I verify which provider was selected for cost reasons?

The dashboard's cost explorer (src/app/(dashboard)/dashboard/costs/costExplorerUtils.ts) uses the same preflightEstimate logic as the router, allowing you to compare estimated costs across providers before sending requests. Additionally, the routing layer logs the selected provider and its corresponding price rate when debug logging is enabled.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}