# How FreeLLMAPI Uses the Thompson-Sampling Bandit Algorithm for Intelligent Model Routing

> Discover how FreeLLMAPI leverages the Thompson-sampling bandit algorithm for intelligent model routing. Optimize API requests by balancing exploration and exploitation of AI models.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-09-04

---

**FreeLLMAPI treats model selection as a contextual bandit problem, using Thompson sampling on a Beta distribution to balance exploration of uncertain models against exploitation of proven performers, while combining this reliability score with speed and intelligence metrics to route each request optimally.**

FreeLLMAPI is an open-source LLM routing layer that dynamically selects the best `(platform, model, key)` tuple for every incoming request. At its core, the system implements a **Thompson-sampling bandit algorithm** to handle the exploration-exploitation trade-off, ensuring high availability while continuously discovering optimal model configurations. The algorithm is primarily implemented in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) and orchestrated through [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts).

## Core Architecture of the Bandit Router

The router evaluates every candidate model using a convex combination of five normalized axes. The **reliability axis** is the probabilistic component where Thompson sampling operates, while the remaining axes provide deterministic signals.

### The Five-Axis Scoring Framework

Each candidate receives a **base score** calculated from weighted components:

```

base = w_rel·reliability + w_spd·speed + w_int·intelligence
effective = base × headroomFactor × rateLimitFactor

```

The **weights** (`w_rel`, `w_spd`, `w_int`) derive from selectable **bandit presets** defined in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) (lines 44-53). Available presets include `balanced`, `smartest`, `fastest`, and `reliable`, allowing operators to prioritize different routing strategies without changing the underlying Thompson-sampling logic.

### Beta Posterior and Decay-Weighted Updates

The reliability score follows a Bayesian updating process. Each model accumulates **decay-weighted** successes and failures over a rolling 7-day window. The system maintains two count arrays per model:

- **Alpha (α)** = successes + community_successes + 1
- **Beta (β)** = failures + community_failures + 1

These parameters define the Beta distribution posterior, implemented in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) (lines 73-82). The addition of `1` to each parameter creates a uniform prior, while optional community-wide counts provide shared prior knowledge across similar models.

## How Thompson Sampling Works in Practice

Thompson sampling solves the contextual bandit by maintaining a probability distribution over the expected reward (reliability) of each model and sampling from these distributions to make decisions.

### Sampling from the Beta Distribution

During live routing, the system draws a random sample from `Beta(α, β)` using the `sampleBeta` function (scoring.ts, lines 88-94). This sample serves as the instantaneous reliability value in the convex combination:

```typescript
import { reliabilityPosterior, sampleBeta } from './scoring';

// Decay-weighted counts for a specific model/key
const successes = 12;
const failures = 3;
const community = { successes: 5, failures: 2 };

const { alpha, beta } = reliabilityPosterior(successes, failures, community);
const reliabilitySample = sampleBeta(alpha, beta);  // Thompson sample

```

The **variance** of the Beta distribution naturally decreases as sample counts increase. Models with few observations maintain high variance, causing the sampler to occasionally select them for exploration. High-performing models with thousands of observations have tight distributions around their true mean, ensuring exploitation of proven reliability.

### Per-Key Bandit Ordering

When a single model has multiple API keys, FreeLLMAPI extends the bandit to the key level. Each key maintains independent success and failure counts, and the router samples a separate Thompson score per key (`sampleBeta` on key-level counts) to determine the optimal key for that specific request (router.ts, lines 388-415). This prevents a single failing key from degrading the perceived reliability of an otherwise healthy model.

## Dynamic Adjustments and Guard Rails

The bandit algorithm operates within a framework of guard rails that respect operational constraints and traffic patterns.

### Peak-Hours Reliability Boosting

During operator-defined peak windows, the router transfers a portion of the speed weight to the reliability weight (scoring.ts, lines 89-95). This **peak-hours adjustment** reinforces the Thompson-sampling effect during high-traffic periods, prioritizing stability over latency when capacity is constrained.

### Task-Type Biasing

The system applies small weight adjustments based on request classification. For `code` versus `chat` tasks, the router shifts marginal weight between axes (scoring.ts, lines 22-28). However, the underlying **Thompson-sampled reliability** remains the stochastic anchor of the scoring function, ensuring that exploration continues regardless of task type.

## Configuration and Presets

Operators can inspect and modify bandit behavior through the preset configuration:

```typescript
import { DEFAULT_STRATEGY, BANDIT_PRESETS } from './scoring';

const strategy = DEFAULT_STRATEGY;  // 'balanced' default
const weights = BANDIT_PRESETS[strategy];
// Returns: { reliability: 0.5, speed: 0.25, intelligence: 0.25 }

```

While the dashboard displays the deterministic expectation `α/(α+β)` for stable sorting (scoring.ts, lines 84-92), the live router always uses the stochastic sample for request distribution.

## Summary

- **Bayesian foundation**: FreeLLMAPI models reliability as a Beta distribution with decay-weighted success/failure counts over a 7-day window.
- **Thompson sampling**: The router draws random samples from `Beta(α, β)` via `sampleBeta` in [`scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/scoring.ts), automatically balancing exploration of uncertain models against exploitation of proven performers.
- **Multi-axis scoring**: Reliability samples combine with deterministic speed and intelligence scores, weighted by selectable presets (`balanced`, `fastest`, `smartest`, `reliable`).
- **Hierarchical bandits**: Per-key Thompson sampling in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) (lines 388-415) ensures optimal API key selection within each model.
- **Dynamic adaptation**: Peak-hours and task-type adjustments shift weight profiles without disrupting the core probabilistic exploration mechanism.

## Frequently Asked Questions

### What is Thompson sampling and why does FreeLLMAPI use it?

Thompson sampling is a Bayesian algorithm for solving multi-armed bandit problems that maintains a probability distribution over the expected reward of each option and samples from these distributions to make decisions. FreeLLMAPI uses it because it provides a principled, probabilistic approach to the exploration-exploitation trade-off, naturally exploring uncertain models (high variance in the Beta distribution) while exploiting reliable ones (low variance) without requiring manual tuning of exploration parameters.

### How does FreeLLMAPI prevent newly added models from being ignored?

New models start with a uniform prior (`α=1, β=1`) plus any community-wide prior counts, resulting in a Beta distribution with high variance. Because Thompson sampling draws random samples from these distributions, models with few observations will occasionally produce high reliability samples due to their wide confidence intervals. This **automatic exploration** ensures new models receive traffic proportional to their uncertainty until sufficient data tightens their posterior distributions.

### Where is the core bandit logic implemented in the codebase?

The mathematical core resides in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts), specifically in the `reliabilityPosterior` function (lines 73-82) for updating Bayesian beliefs and the `sampleBeta` function (lines 88-94) for stochastic sampling. The orchestration logic that applies these scores to routing decisions lives in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) (lines 388-415), which handles per-key bandit ordering and the final model selection chain.

### Can I disable exploration and always pick the most reliable model?

While you cannot disable Thompson sampling entirely without code modification, you can minimize exploration by selecting the `reliable` preset, which maximizes the weight of the reliability axis. Additionally, the system uses decay-weighted counts, so as models accumulate thousands of observations, their Beta distributions become extremely tight (low variance), causing the sampler to behave nearly deterministically. For complete determinism, you would need to modify `sampleBeta` in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) to return the mean `α/(α+β)` instead of a random sample.