How the FreeLLMAPI Router Selects the Best Available LLM Model: Bandit Scoring and Fallback Chains

The FreeLLMAPI router selects the best available LLM model by computing a bandit-style convex score from reliability, speed, and intelligence signals, applying guard-rail multipliers for free-tier headroom and live rate-limit penalties, then choosing the highest-scoring model with a usable API key from an ordered fallback chain.

The tashfeenahmed/freellmapi repository implements an intelligent request router that automatically selects the optimal large language model for each request. Understanding how the FreeLLMAPI router selects the best available LLM model requires examining its three-stage pipeline: chain resolution, bandit scoring, and key validation. This architecture makes the router both data-driven (learning from historical successes and failures) and policy-driven (allowing operators to tune selection via routing strategies).

Building the Candidate Chain

The router first constructs an ordered list of candidate models through a fallback chain resolution process. This stage determines which models are eligible for scoring based on user configuration and global defaults.

Profile-Aware Fallback Resolution

The router inspects the active user profile to determine the candidate pool. If a user has an active profile, the system pulls models linked to that profile via the profile_models association; otherwise, it falls back to the global fallback_config table. This logic resides in server/src/services/router.ts within the getActiveChain function (lines 26-55).

function getActiveChain(db: Database): ChainRow[] { … }

The resolveRoutingChain function (lines 14-22) handles dynamic shortcuts and global sort aliases, ensuring the router adapts to both personalized and default routing configurations.

Dynamic Global Sort Aliases

Special call patterns like auto:fast are translated into preset strategies (such as fastest) via the GLOBAL_SORT_ALIASES mapping. When triggered, the router builds a chain of all enabled models and orders them according to the alias-defined strategy. This allows users to request speed-optimized routing without explicitly naming a model.

Bandit-Style Model Scoring

Once the candidate chain is resolved, the router scores each model using a bandit algorithm that balances exploitation of known high performers with exploration of newer options. The orderChain function (lines 76-95 of router.ts) orchestrates this process, calling scoreChainEntry and combineScore to compute final rankings.

Normalized Signal Axes

Each model is evaluated across three normalized axes, defined in server/src/services/scoring.ts:

• Reliability: Computed via a Beta posterior of success/failure counts using reliabilityPosterior (lines 31-40). For live routing, this uses Thompson sampling (sampleBeta) to inject exploration, while dashboard views use expectedReliability for stable averages. Returns a value in [0, 1].

• Speed: The speedScore function blends throughput (throughputScore) and first-byte latency (ttfbScore), returning [0, 1] or the SPEED_PRIOR if no historical data exists (lines 1-7).

• Intelligence: The intelligenceComposite function calculates a tier-first composite score normalized by min/max values across the chain, returning [0, 1] (lines 12-15).

Guard-Rail Multipliers

Before finalizing scores, the router applies two guard-rail multipliers to prevent over-utilization of constrained resources:

• Headroom Factor: The headroomFactor function (lines 24-30 of scoring.ts) reduces the score as a model approaches its free-tier token budget, ensuring proportional routing away from quota-limited providers.

• Rate-Limit Penalty: Every HTTP 429 response increments a per-model penalty via recordRateLimitHit (lines 82-90 of router.ts). The penalty decays over time and is converted into a multiplier by rateLimitFactor (lines 32-42 of scoring.ts), temporarily suppressing models that are currently throttled.

The Convex Combination Formula

The combineScore function (lines 85-92 of scoring.ts) calculates the final score using a weighted convex combination of the three signal axes, then applies the guard-rail multipliers:

export function combineScore(inputs: ScoreInputs, weights: RoutingWeights): number {
  const wSum = weights.reliability + weights.speed + weights.intelligence || 1;
  const base = (weights.reliability * inputs.reliability +
                weights.speed * inputs.speed +
                weights.intelligence * inputs.intelligence) / wSum;
  return base * inputs.headroom * inputs.rateLimit;
}

The weights parameter is determined by the active routing strategy (e.g., balanced, fastest, reliable), allowing operators to bias selection toward specific model characteristics.

API Key Selection and Validation

After sorting the chain by score descending, the router iterates through the ordered list to find a usable model. For each candidate, selectKeyForModel (lines 53-66 of router.ts) performs the following validation steps:

1. Verifies the provider exists via hasProvider.
2. Retrieves all enabled and healthy keys for the platform.
3. Validates per-key constraints: cooldown status, provider daily caps, RPM/RPD limits, TPM/TPD limits, and decryption success.
4. Rotates selection across valid keys using a roundRobinIndex.

If a key passes all gates, the router returns a RouteResult containing the provider instance, decrypted API key, and model identifiers. If no key is available for a given model, the router proceeds to the next entry in the scored chain. When the entire chain is exhausted, the router throws a RouteError with status 429, containing diagnostics explaining why each model was rejected (lines 19-28).

Practical Implementation Examples

Retrieve the Ordered Model Chain

To inspect the current model ranking without making a request, use getOrderedFusionChain:

import { getOrderedFusionChain } from './services/router';

const chain = getOrderedFusionChain();   // Returns FusionCandidate[]
console.log('Top-scoring models:', chain.slice(0, 3));

This returns an array of objects including modelDbId, platform, modelId, and displayName, already sorted by the active strategy.

Switch Routing Strategies Programmatically

Force the router to prioritize speed over reliability by changing the strategy:

import { setRoutingStrategy, getRoutingStrategy } from './services/router';

setRoutingStrategy('fastest');
console.log('Active strategy:', getRoutingStrategy()); // → 'fastest'

The next call to routeRequest will use the weight vector defined in BANDIT_PRESETS for the fastest configuration.

Pin a Specific Model for Sticky Sessions

To ensure a conversation consistently uses a specific model (e.g., database ID 42), pass the model ID to routeRequest:

import { routeRequest } from './services/router';

const result = routeRequest(1200, undefined, 42);
console.log(`Pinned ${result.modelId} on ${result.platform}`);

The router moves the pinned model to the front of the sorted chain while still executing the standard key-selection validation.

Summary

• Three-stage pipeline: The router resolves the active chain from profile or global config, scores models using a bandit algorithm, and selects the first model with a valid API key.
• Bandit scoring: Combines reliability (Beta posterior), speed (throughput + latency), and intelligence (tier composite) into a weighted convex score, with Thompson sampling for exploration.
• Guard-rails: Applies headroomFactor for free-tier quotas and rateLimitFactor for live 429 penalties to prevent routing to constrained models.
• Strategy-driven: Uses configurable RoutingWeights (balanced, fastest, reliable) to bias selection without hardcoding model preferences.
• Fallback mechanism: Automatically proceeds down the scored chain if a model's keys are exhausted, cooldown-limited, or capped, ensuring high availability.

Frequently Asked Questions

What routing strategies does FreeLLMAPI support?

FreeLLMAPI supports multiple routing strategies defined in BANDIT_PRESETS, including balanced (default), fastest, and reliable. Each strategy defines a specific weight vector for the convex combination of reliability, speed, and intelligence signals. Operators can switch strategies dynamically via setRoutingStrategy(), and users can trigger preset aliases like auto:fast which map to these configurations.

How does the router handle rate limits and 429 errors?

When a provider returns a 429 status, the router invokes recordRateLimitHit (lines 82-90 of router.ts) to increment a per-model penalty. This penalty decays over time and is applied as a multiplicative factor via rateLimitFactor (lines 32-42 of scoring.ts), temporarily reducing the model's selection score. The router also checks hard limits including RPM, RPD, TPM, and TPD via canMakeRequest in ratelimit.ts before allowing a key to be used.

Can I force the router to use a specific model instead of automatic selection?

Yes. Pass the specific modelDbId as the third argument to routeRequest(). The router will move that model to the front of the sorted chain via the pinning logic, then execute the standard selectKeyForModel validation. If the pinned model has no usable keys, the router falls back to the next model in the chain unless configured otherwise.

What happens when all models in the fallback chain are exhausted?

If selectKeyForModel fails to find a valid key for every model in the ordered chain, the routeRequest function throws a RouteError with HTTP status 429 (lines 19-28 of router.ts). The error includes detailed diagnostics indicating why each model was rejected—such as cooldown status, decryption failure, or quota exhaustion—enabling operators to diagnose systemic provider issues.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}