How Model Scores Are Calculated in FreeLLMAPI: A Deep Dive into the Routing Engine

FreeLLMAPI calculates model scores using a multi-dimensional weighted system that combines reliability, speed, and intelligence metrics into a normalized ranking between 0 and 1, then applies Thompson sampling and configurable overrides to determine routing priority.

FreeLLMAPI is an open-source LLM routing platform that dynamically selects the best available model for each request. Understanding how model scores are calculated in FreeLLMAPI requires examining the scoring pipeline implemented in the Node.js/TypeScript backend. The system evaluates every model across three quantitative axes, applies operator-defined weight profiles, and uses bandit algorithms to balance exploration with exploitation.

The Three Core Axes of FreeLLMAPI Scoring

The scoring engine in server/src/services/scoring.ts derives each model's base performance from three independent metrics normalized to the [0, 1] range.

Reliability Metrics

Reliability represents the posterior probability of successful request completion based on historical success/failure statistics. The system aggregates community-reported outcomes and internal health checks to compute a Bayesian probability that a given model will respond without errors. According to the source implementation, this calculation occupies lines 213-224 in scoring.ts, where the system updates belief distributions based on recent observation windows while maintaining a minimum sample threshold to prevent volatile early-stage models from dominating the rankings.

Speed Measurements

Speed quantifies throughput efficiency through a composite of tokens-per-second throughput and Time-To-First-Byte (TTFB) measurements. The normalization logic in scoring.ts (lines 258-313) blends these latency indicators into a single "speed score" using percentile rankings against recent historical performance. This ensures that models are evaluated relative to current network conditions rather than absolute benchmarks that might become outdated as infrastructure evolves.

Intelligence Weights

Intelligence serves as a static calibration factor defined per-model in the configuration layer. Newer or larger parameter models receive higher base intelligence scores, typically ranging from 0.7 for legacy endpoints to 1.0 for state-of-the-art releases. As defined in lines 46-53 of scoring.ts, these weights remain constant during runtime but can be updated via configuration deployments to reflect new benchmark releases or capability assessments.

Weight Profiles and the Base Score Calculation

FreeLLMAPI implements a convex combination formula to merge the three axes into a unified base score:

baseScore = (w_rel × reliability) + (w_spd × speed) + (w_int × intelligence)

Preset Routing Profiles

The system ships with four preset weight profiles defined in scoring.ts (lines 89-111):

• Balanced: Equal weighting (0.33/0.33/0.33) across all three axes for general-purpose routing
• Smartest: Prioritizes intelligence (0.6) with reduced emphasis on speed (0.2) and reliability (0.2)
• Fastest: Maximizes speed weight (0.7) while minimizing intelligence (0.15) and reliability (0.15) for latency-sensitive applications
• Reliable: Favors reliability (0.6) over speed (0.2) and intelligence (0.2) for critical production workloads

Operators can dynamically shift profiles during peak traffic hours. The implementation allows real-time migration of the speed weight partially onto reliability to guard against flaky fast models when infrastructure is under stress.

Advanced Scoring Mechanisms

Beyond the base calculation, FreeLLMAPI applies several corrective layers to refine routing decisions.

Model-Specific Overrides

Administrators may define per-model multipliers via the MODEL_ROUTING_OVERRIDES configuration object implemented in server/src/services/model-weight-overrides.ts (lines 17-33). These scalar values multiply the base score after the initial weighted combination, enabling fine-grained priority adjustments without disrupting the global weighting strategy. For example, a model with a 1.2 override multiplier receives a 20% boost to its final routing score regardless of the active profile.

Bandit Scoring and Thompson Sampling

The routing layer in server/src/services/router.ts (lines 921-1020) implements Thompson Sampling to handle the exploration-exploitation trade-off. Each model maintains a per-key bandit distribution sampled during the selection phase. When multiple models fall within the same performance tier, the system samples from these distributions to introduce stochastic exploration while still respecting the deterministic expected score for display and monitoring purposes. This prevents the system from prematurely converging on suboptimal models due to early variance in observations.

Final Score Clamping and Fallback Protection

Before finalizing rankings, the combined score undergoes clamping operations defined in scoring.ts (lines 455-458). The system enforces a minimum floor—typically 10% of the original calculated score—to ensure degraded models remain visible in the fallback chain. This prevents total blackouts when primary models experience widespread failures, allowing the router to gracefully cascade to secondary options even if their raw metrics have deteriorated significantly.

Accessing Model Scores via the API

FreeLLMAPI exposes calculated scores through two primary endpoints for integration and monitoring.

The MCP Endpoint

The GET /mcp route defined in server/src/routes/mcp.ts (lines 163-172) returns the active routing strategy and an array of top-scored models. Each entry includes the model identifier, platform source, and calculated score rounded to three decimal places. Client applications can query this endpoint to implement client-side caching or display availability dashboards.

The Routing Diagnostics Endpoint

For human-readable diagnostics, the GET /routing endpoint in server/src/services/router.ts (lines 2075-2092) provides detailed breakdowns of how each model's score was derived, including the raw reliability, speed, and intelligence components before weighting.

Fetching Scores Programmatically

import fetch from 'cross-fetch';

async function getTopModels() {
  const resp = await fetch('https://api.freellmapi.com/mcp');
  const { scores } = await resp.json();
  
  // scores: [{ model: string, platform: string, score: number }, ...]
  return scores
    .sort((a, b) => b.score - a.score)
    .slice(0, 5);
}

async function chatWithBestModel(message: string) {
  const topModel = (await getTopModels())[0];
  const endpoint = `https://api.freellmapi.com/${topModel.model}/chat`;
  
  const reply = await fetch(endpoint, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      messages: [{ role: 'user', content: message }]
    }),
  });
  
  return reply.json();
}

Summary

• Three-axis foundation: FreeLLMAPI calculates model scores using reliability (success probability), speed (throughput/TTFB), and intelligence (static capability weight) metrics normalized to [0, 1].
• Configurable weighting: The system applies preset profiles (balanced, smartest, fastest, reliable) that determine the convex combination weights for each axis.
• Override capabilities: Per-model multipliers in MODEL_ROUTING_OVERRIDES allow administrators to manually adjust specific model priorities without changing global algorithms.
• Stochastic routing: Thompson Sampling in the router layer introduces controlled exploration to prevent premature convergence on suboptimal models.
• API transparency: Scores are exposed via GET /mcp for programmatic access and GET /routing for operational diagnostics, both returning three-decimal precision rankings.

Frequently Asked Questions

What file contains the core scoring logic for FreeLLMAPI?

The primary scoring implementation resides in server/src/services/scoring.ts. This file contains the calculation logic for reliability posterior probabilities, speed normalization, intelligence weight assignments, and the final weighted combination formulas used to generate base scores.

How does FreeLLMAPI prevent fast but unreliable models from dominating the routing?

The system implements dynamic weight profile adjustments that can shift priority from speed to reliability during peak hours or degraded conditions. Additionally, the clamping mechanism in the scoring pipeline ensures models with poor reliability statistics receive reduced but non-zero scores, keeping them in the fallback chain without allowing them to capture primary traffic.

Can I use the FreeLLMAPI scoring system to evaluate models on my own infrastructure?

Yes. While the hosted service manages global reliability statistics, the open-source codebase in the tashfeenahmed/freellmapi repository allows self-hosted deployments to configure local reliability tracking, custom weight profiles, and private model overrides. The GET /mcp endpoint provides the same score transparency for custom deployments as the managed service.

What is the difference between base scores and bandit scores in FreeLLMAPI?

Base scores represent the deterministic weighted combination of reliability, speed, and intelligence metrics. Bandit scores incorporate Thompson Sampling to introduce stochastic variance, allowing the system to occasionally test lower-ranked models for potential improvement. The router uses bandit scores for actual selection decisions while displaying base scores for monitoring and diagnostics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}