How FreeLLMAPI Uses Thompson-Sampling for Intelligent LLM Routing

FreeLLMAPI routes requests to language model providers using Thompson-sampling, a multi-armed bandit algorithm that maintains Beta distributions for each provider key and samples from these distributions to balance exploration and exploitation.

FreeLLMAPI is an open-source routing layer that intelligently distributes traffic across multiple LLM providers. At the heart of its decision-making process lies a Thompson-sampling bandit implemented in TypeScript, which continuously adapts routing decisions based on real-time success and failure signals. This approach ensures optimal provider selection without requiring manual configuration changes when provider performance fluctuates.

The Beta Posterior Scoring Engine

The foundation of FreeLLMAPI's routing intelligence resides in server/src/services/scoring.ts. As noted in the source at line 19, the system treats each provider key's reliability as a random variable drawn from a Beta posterior distribution.

For every key, the service tracks two counters:

• Successes (α): Incremented when a request completes without rate-limit errors or failures
• Failures (β): Incremented when a request encounters errors or timeouts

The posterior distribution becomes Beta(α + 1, β + 1), where the added one represents an uninformative prior. When the router queries for scores, it draws a random sample from each key's Beta distribution. This sampling mechanism naturally balances exploitation—selecting keys with historically high success rates—against exploration—giving newer or temporarily failing keys a probabilistic chance to prove themselves.

Request Routing with Thompson-Sampling

The routing orchestration in server/src/services/router.ts implements the Thompson-sampling selection logic. According to line 1018, the sampled parameter controls the bandit branch, with Thompson-sampling serving as the default strategy.

Key implementation details include:

• Line 388: The exploration toggle activates the bandit. When enabled, "the bandit's Thompson sampling already explores," eliminating the need for separate epsilon-greedy exploration.

• Line 921: The sampled flag explicitly chooses Thompson-sampling over deterministic ordering.

• Line 1275: The router "orders a model's candidate keys by a Thompson-sampled per-key score, mirroring" the bandit's preference mechanism.

• Line 538: The system prevents excessive historical weighting because "count would drown local evidence forever and collapse the Thompson-sampling," ensuring the algorithm remains responsive to recent performance changes.

When a request arrives, the router calls the scoring service with sampled: true, retrieves the sampled values for all candidate keys, and selects the key with the highest sampled score for the next hop.

Architecture and Strategy Presets

The high-level architecture documented in docs/architecture.md (line 49) describes this mechanism as the "smartest" routing preset. According to the documentation, "scores come from live per-model measurements ... with a Thompson-sampling bandit under the hood."

Users can toggle between routing strategies:

• Smartest (default): Uses Thompson-sampling with Beta posteriors
• Priority: Disables sampling and uses deterministic manual ordering

This architectural flexibility allows operators to enforce strict provider hierarchies when needed while benefiting from adaptive routing during normal operations.

Implementation Examples

The following examples demonstrate how to interact with FreeLLMAPI's Thompson-sampling router programmatically and via CLI.

Importing and Using the Scoring Service

To manually evaluate candidate keys using Thompson-sampling:

import { getScoredKeys } from '@freellmapi/server/src/services/scoring';

const model = 'gpt-4';
const candidateKeys = ['openrouter-abc', 'groq-def', 'huggingface-ghi'];

// The sampled: true flag enables Thompson-sampling
const scored = await getScoredKeys(model, candidateKeys, { sampled: true });

/*
scored = [
  { key: 'groq-def', score: 0.87 },      // Highest sample selected
  { key: 'openrouter-abc', score: 0.45 },
  { key: 'huggingface-ghi', score: 0.12 },
]
*/

Enabling Smart Routing via CLI

To activate the Thompson-sampling bandit from the command line:

freellmapi run --model gpt-4 --strategy smartest

Disabling Exploration for Deterministic Routing

For testing or strict priority requirements, disable Thompson-sampling:

freellmapi run --model gpt-4 --strategy priority

Summary

• FreeLLMAPI implements Thompson-sampling via Beta posterior distributions in server/src/services/scoring.ts to model provider reliability.
• The router in server/src/services/router.ts uses the sampled parameter (defaulting to true) to trigger probabilistic key selection based on drawn Beta samples.
• Each provider key maintains α (success) and β (failure) counters that update after every request, ensuring the routing logic adapts to real-time conditions.
• The "smartest" strategy enables the bandit, while the "priority" strategy disables sampling for deterministic routing.
• Line 538 in router.ts contains critical logic preventing historical counts from overwhelming recent evidence, preserving the bandit's responsiveness.

Frequently Asked Questions

What is Thompson-sampling in FreeLLMAPI?

Thompson-sampling is a probabilistic algorithm that treats provider selection as a multi-armed bandit problem. FreeLLMAPI maintains a Beta distribution representing the reliability belief for each provider key in server/src/services/scoring.ts. For each routing decision, the system samples a random value from each key's distribution and selects the key with the highest sample. This method naturally balances exploring lesser-used providers against exploiting known reliable ones.

How does FreeLLMAPI update provider reliability scores?

After every request, the system updates the Beta posterior parameters stored in server/src/services/scoring.ts. A successful request increments the α (success) parameter, while failures or rate-limits increment the β (failure) parameter. The scoring service uses these updated values to generate new samples for subsequent routing decisions, creating a continuous feedback loop that responds to provider performance changes within seconds.

What is the difference between the "smartest" and "priority" routing strategies?

The "smartest" strategy activates FreeLLMAPI's Thompson-sampling bandit according to line 1018 in server/src/services/router.ts, allowing the system to probabilistically route traffic based on observed success rates. The "priority" strategy disables sampling entirely and routes requests according to a static, manually-ordered list of providers. Use "smartest" for production workloads requiring automatic failover and load balancing, and "priority" when you need deterministic routing for testing or compliance requirements.

Where is the Thompson-sampling logic tested in the codebase?

The Beta sampler and Thompson-sampling behavior are validated in server/src/__tests__/services/scoring.test.ts. These tests ensure that the posterior updates, sampling distributions, and exploration characteristics behave as expected under various success and failure scenarios, confirming that the bandit correctly balances exploration and exploitation across provider keys.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}