FreeLLMAPI Routing Scoring Strategies: Bandit, Reliability, and Balanced Presets Explained

FreeLLMAPI uses a bandit-style convex combination of reliability, speed, and intelligence scores, with configurable weight presets including BALANCED, RELIABLE, and SMARTEST, to route requests to the optimal language model.

FreeLLMAPI implements a sophisticated multi-armed bandit scoring system to intelligently route API requests across different language models. The routing engine in tashfeenahmed/freellmapi evaluates every enabled model along three normalized axes—reliability, speed, and intelligence—then applies user-configurable weight strategies to select the highest-scoring candidate. This approach optimizes for both performance and accuracy while respecting free-tier quota constraints.

The Three Scoring Axes in FreeLLMAPI

The scoring pipeline in server/src/services/scoring.ts calculates a base score from three normalized dimensions before applying guard-rail factors.

Reliability Scoring via Beta Posterior

Reliability measures a model's success probability using Bayesian inference with a Beta distribution. The system maintains success and failure counts for each model, then calculates the posterior parameters:

// server/src/services/scoring.ts (lines 59-66)
export function reliabilityPosterior(successes: number, failures: number) {
  const alpha = 1 + successes;
  const beta = 1 + failures;
  return { alpha, beta };
}

export function expectedReliability(successes: number, failures: number) {
  const { alpha, beta } = reliabilityPosterior(successes, failures);
  return alpha / (alpha + beta);
}

For Thompson sampling, the system draws a random sample from the Beta distribution using sampleBeta (line 64), enabling exploration of potentially high-performing models with limited historical data.

Speed Score Calculation

Speed combines throughput (tokens per second) and time-to-first-byte (TTFB) into a single metric. The speedScore function (line 101) applies an exponential-saturating curve to throughput and a linear ramp to TTFB, then blends them with a 60/40 weighting:

// 0.6 * throughputScore + 0.4 * ttfbScore

This ensures models with both high throughput and low latency receive optimal scores.

Intelligence Tier Normalization

Intelligence captures relative model capability across tiers (Frontier → Large → Medium → Small) plus intra-tier rankings. The intelligenceScore function (line 112) min-max normalizes these composite values to the range [0, 1], allowing direct comparison between different model classes.

Available Routing Strategies and Weight Presets

FreeLLMAPI provides six routing strategies that determine which weight vector applies to the three scoring axes. These presets are defined in the BANDIT_PRESETS constant (lines 36-46) of server/src/services/scoring.ts.

BALANCED Strategy

The default strategy (DEFAULT_STRATEGY at line 51) assigns equal emphasis to reliability while splitting the remaining weight between speed and intelligence:

balanced: { reliability: 0.5, speed: 0.25, intelligence: 0.25 }

This configuration provides a middle ground suitable for general-purpose workloads.

RELIABLE Strategy

For production workloads requiring maximum uptime, the reliable preset heavily weights reliability:

reliable: { reliability: 0.7, speed: 0.15, intelligence: 0.15 }

This strategy prioritizes models with proven track records, accepting potential trade-offs in speed or capability.

FASTEST and SMARTEST Strategies

• fastest: { reliability: 0.35, speed: 0.55, intelligence: 0.10 } — Optimizes for throughput and low latency while maintaining a reliability floor.
• smartest: { reliability: 0.35, speed: 0.10, intelligence: 0.55 } — Favors frontier-tier models for complex reasoning tasks.

PRIORITY and CUSTOM Strategies

• priority: Legacy manual ordering that bypasses bandit scoring entirely, using only the 429 penalty factor (handled in router.ts line 77).
• custom: User-defined weights persisted in settings and normalized on save via setCustomWeights().

Guard-Rail Factors and Score Combination

The final score combines the weighted base score with two multiplicative guard-rail factors that can only decrease the result. The combineScore function (lines 85-92) performs the convex combination:

export function combineScore(inputs: ScoreInputs, weights: RoutingWeights): number {
  const wSum = weights.reliability + weights.speed + weights.intelligence || 1;
  const base =
    (weights.reliability * inputs.reliability +
      weights.speed * inputs.speed +
      weights.intelligence * inputs.intelligence) / wSum;
  return base * inputs.headroom * inputs.rateLimit;
}

Headroom Factor

The headroom factor protects models nearing their free-tier token quota. Defined in lines 124-130 with HEADROOM_RAMP_START = 0.2 and HEADROOM_FLOOR = 0.1, it returns 1 when ≥20% of the budget remains, then linearly decays to 0.1 when exhausted.

Rate-Limit Factor

The rate-limit factor demotes throttled models (429 responses). With MAX_PENALTY = 10 and RATE_LIMIT_MAX_DAMP = 0.6 (lines 136-142), the factor ranges from 1 (no penalty) down to 0.4 (maximum penalty).

Implementing Custom Routing Logic

You can programmatically inspect and modify routing strategies using the router service:

import {
  getRoutingStrategy,
  setRoutingStrategy,
  getCustomWeights,
  setCustomWeights,
  BANDIT_PRESETS,
} from '@/server/src/services/router';

// Query current strategy
const current = getRoutingStrategy(); // 'balanced' | 'reliable' | ...

// Switch to fastest preset
setRoutingStrategy('fastest');
console.log('Weights:', BANDIT_PRESETS.fastest);

// Set custom weights (automatically normalized)
setCustomWeights({ reliability: 0.7, speed: 0.2, intelligence: 0.1 });

To compute scores manually for dashboards or testing:

import {
  reliabilityPosterior,
  sampleBeta,
  speedScore,
  intelligenceScore,
  combineScore,
} from '@/server/src/services/scoring';

function computeModelScore(
  successes: number,
  failures: number,
  tokPerSec: number,
  ttfbMs: number,
  tierRank: number,
  tierMin: number,
  tierMax: number,
  quotaUsed: number,
  quotaBudget: number,
  ratePenalty: number,
  strategy: 'balanced' | 'reliable' | 'fastest' | 'smartest' | 'custom',
  customWeights?: { reliability: number; speed: number; intelligence: number }
) {
  // Reliability via Thompson sampling
  const posterior = reliabilityPosterior(successes, failures);
  const rel = sampleBeta(posterior.alpha, posterior.beta);
  
  // Speed and intelligence
  const spd = speedScore(tokPerSec, ttfbMs);
  const intel = intelligenceScore(tierRank, tierMin, tierMax);
  
  // Guard-rails
  const headroom = quotaUsed / quotaBudget > 0.8 ? 0.1 + 0.9 * (1 - quotaUsed/quotaBudget)/0.2 : 1;
  const rateLimit = Math.max(0.4, 1 - (ratePenalty / 10) * 0.6);
  
  // Select weights
  const weights = strategy === 'custom' ? customWeights : BANDIT_PRESETS[strategy];
  
  return combineScore(
    { reliability: rel, speed: spd, intelligence: intel, headroom, rateLimit },
    weights
  );
}

Summary

• FreeLLMAPI evaluates models on three normalized axes: reliability (Beta posterior), speed (throughput + TTFB), and intelligence (capability tier).
• The BANDIT_PRESETS in server/src/services/scoring.ts provide five preconfigured strategies: balanced (default), reliable, fastest, smartest, and priority (legacy).
• Guard-rail factors (headroom and rate-limit) multiplicatively reduce scores for quota-constrained or throttled models, ensuring stable routing.
• The combineScore function performs a convex combination of weighted inputs, always normalizing user-supplied weights to sum to 1.
• Custom strategies can be persisted via setCustomWeights() and setRoutingStrategy() in server/src/services/router.ts.

Frequently Asked Questions

What is the difference between the RELIABLE and BALANCED scoring strategies?

The RELIABLE strategy assigns 70% weight to the reliability axis, ensuring maximum uptime by favoring models with proven success records, while BALANCED uses a 50/25/25 split across reliability, speed, and intelligence. Use RELIABLE for production workloads where consistency matters more than speed, and BALANCED for general-purpose applications.

How does FreeLLMAPI handle models that are close to their token quota?

The system applies a headroom factor (implemented in server/src/services/scoring.ts lines 124-130) that linearly reduces a model's score when it drops below 20% of its free-tier budget. At 0% remaining quota, the factor reaches its floor of 0.1, effectively deprioritizing the model while allowing occasional fallback requests.

Can I create a completely custom scoring weight configuration?

Yes. Set the strategy to custom using setRoutingStrategy('custom'), then persist your weights with setCustomWeights({ reliability, speed, intelligence }). The router automatically normalizes these values so they sum to 1, and stores them in the settings table for persistence across restarts.

Does the BANDIT preset refer to a specific algorithm implementation?

The term BANDIT refers to the Thompson sampling approach used in the reliability calculation (drawing from a Beta posterior via sampleBeta), combined with the convex weighting scheme across multiple objectives. All non-priority strategies use this multi-armed bandit framework to balance exploration (trying newer models) with exploitation (using proven high-performers).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}