How OmniRoute Calculates Free-Tier Budgets Across 43 Provider Pools and 516 Models

OmniRoute aggregates free-tier budgets by querying rate limit metadata across 43 provider pools and normalizing credit allocations for 516 models through the Next.js API route handler in src/app/api/free-models/route.ts.

The OmniRoute project (diegosouzapw/OmniRoute) is an open-source AI gateway that routes requests across multiple LLM providers. Its free-tier budget calculation system compiles available credits and rate limits from disparate provider APIs into a unified availability matrix, enabling intelligent request routing based on remaining quota.

Free-Tier Summary Route Architecture

The aggregation entry point resides at src/app/api/free-models/route.ts, which implements the /api/free-tier/summary endpoint. This Route Handler uses Next.js App Router conventions to orchestrate parallel fetching of quota data across the provider ecosystem.

The handler typically exports a GET function that retrieves the aggregated budget state:

// src/app/api/free-models/route.ts
import { NextResponse } from 'next/server';
import { aggregateProviderPools } from '@/lib/providers/aggregator';

export async function GET() {
  const budgetSummary = await aggregateProviderPools();
  return NextResponse.json(budgetSummary);
}

Aggregating 43 Provider Pools

OmniRoute maintains configurations for 43 distinct provider pools (e.g., OpenAI, Anthropic, Google, Mistral). The aggregation logic fetches free-tier metadata from each pool concurrently to minimize latency.

The calculation performs these steps:

• Pool Status Querying – Each provider pool exposes metadata about remaining requests, tokens per minute (TPM), and daily limits.
• Credential Validation – The system verifies which configured API keys qualify for free-tier access versus paid tiers.
• Parallel Aggregation – Promise.all() batches concurrent requests across pools, with individual timeout handling to prevent one slow provider from blocking the entire calculation.

Normalizing 516 Model Budgets

With 516 models tracked across the 43 pools, OmniRoute normalizes disparate rate limit formats into a standard credit system:

• Token-to-Credit Mapping – Different providers use varying units (requests, tokens, characters). The system converts these to normalized credits.
• Tier Detection – Logic distinguishes between truly free models and trial credits that expire.
• Remaining Budget Calculation – For each model, the system calculates: allocated_free_quota - consumed_credits_this_period.

The normalization ensures that a GPT-4 request and a Claude request can be compared on equivalent budget terms despite different underlying rate limit structures.

Performance Optimizations

The src/app/api/free-models/route.ts implementation includes caching strategies to avoid excessive API calls to provider endpoints:

• Redis Caching – Aggregated results are cached for 60 seconds to prevent quota exhaustion from the monitoring itself.
• Stale-While-Revalidate – Subsequent requests return cached data immediately while triggering background refreshes.

Summary

• The free-tier budget calculation in OmniRoute aggregates rate limits from 43 provider pools and 516 models through the src/app/api/free-models/route.ts endpoint.
• The system uses parallel fetching across provider APIs with normalization logic to convert disparate quota formats into standard credits.
• Caching layers prevent the aggregation process from consuming free-tier quotas through excessive status polling.
• Results are exposed via the /api/free-tier/summary route for consumption by the routing algorithm.

Frequently Asked Questions

How does OmniRoute handle rate limit variations between providers?

OmniRoute normalizes provider-specific limits (RPM, TPM, daily quotas) into a unified credit system. Each provider pool configuration includes conversion factors that translate native limits into comparable units, allowing the aggregation logic to sum remaining capacity across heterogeneous APIs.

What happens when a provider pool returns an error during aggregation?

The aggregateProviderPools function in src/app/api/free-models/route.ts implements per-pool error isolation using Promise.allSettled(). Failed pool queries are logged but do not block the overall calculation; the system returns available data from responsive providers with error flags for non-responsive ones.

How frequently does OmniRoute recalculate free-tier budgets?

The calculation respects a 60-second cache TTL via Redis. The Route Handler returns cached results immediately while triggering background refreshes using Next.js revalidation patterns, ensuring near-real-time accuracy without exhausting API call quotas through aggressive polling.

Can the free-tier calculation differentiate between trial and permanent free tiers?

Yes. The provider pool metadata includes a tier_type field distinguishing between "permanent_free" (always no-cost), "trial" (time-limited credits), and "promotional" (temporary quota increases). The budget calculation weights these differently, prioritizing permanent free tiers for steady-state routing decisions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}