# How FreeLLMAPI's Router Selects Models and Handles Automatic Fallback

> Discover how FreeLLMAPI's router selects models and handles automatic fallback by dynamically scoring and ranking them based on reliability speed and intelligence for seamless API requests.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-28

---

**FreeLLMAPI routes every request through a dynamically scored fallback chain that ranks enabled models by reliability, speed, and intelligence, then performs automatic fallback to the next candidate if API keys are exhausted or rate-limited.**

The open-source FreeLLMAPI project (tashfeenahmed/freellmapi) implements a sophisticated routing layer that automatically selects the optimal large language model for each request. Located primarily in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) and [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts), the system combines multi-armed bandit algorithms with real-time availability checks to build a prioritized fallback chain. This ensures high availability by gracefully degrading to alternative models when primary options fail.

## Building the Active Fallback Chain

The routing process begins by fetching the current configuration from the database. According to the tashfeenahmed/freellmapi source code, the function `getActiveChain(db)` retrieves either a profile-specific chain or the global fallback configuration, returning an ordered list of enabled models and their associated API keys.

```typescript
const chain = getActiveChain(db);  // server/src/services/router.ts

```

This chain represents the complete pool of available models. The router evaluates this list fresh on every request to ensure it reflects the latest state, including newly added keys, updated quotas, or changed routing strategies.

## Bandit Scoring and Routing Strategies

Once the chain is loaded, the `orderChain(chain, strategy)` function applies the configured routing strategy to score and sort every model. FreeLLMAPI supports six built-in strategies: **priority**, **balanced**, **smartest**, **fastest**, **reliable**, and **custom** user-defined weight vectors.

The scoring engine in [`server/src/services/scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/scoring.ts) calculates raw scores using a bandit algorithm. **Reliability** is computed from a Beta posterior distribution using Thompson sampling for live routing, while dashboard displays use the deterministic expected value. **Speed** combines throughput metrics with first-byte latency. **Intelligence** uses a tier-based ranking system. Two guard-rails—a head-room check and a rate-limit penalty—demote models nearing quota or currently returning HTTP 429 errors.

```typescript
const sortedChain = orderChain(chain, strategy);  // server/src/services/router.ts

```

The sorted chain places the highest-scoring candidate first, ensuring the "best" available model is attempted according to the chosen strategy.

## API Key Selection and Round-Robin Logic

For each candidate model in the sorted chain, the router executes `selectKeyForModel(entry, estimatedTokens, skipKeys, diag)` to choose a specific API key. This function, also defined in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), implements round-robin selection across all enabled keys for that model.

The selection logic performs several critical checks:

- **Cooldown status**: Keys in an active cooldown period are skipped
- **Rate limits**: Per-minute and per-day quotas are validated against [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) functions
- **Decryption failures**: Keys that cannot be decrypted are bypassed
- **Token capacity**: The `estimatedTokens` parameter ensures the key’s remaining quota can handle the request

The first usable key encountered becomes the **RouteResult**. If no key passes these checks, the function returns `null`, triggering the fallback mechanism.

## Automatic Fallback Execution

Automatic fallback occurs when `selectKeyForModel` returns `null` or when a downstream error forces a retry. The router implements this as a sequential loop over the `sortedChain`:

```typescript
for (const entry of sortedChain) {
  // Vision, tools, and context capability checks occur here
  const route = selectKeyForModel(entry, estimatedTokens, skipKeys, diag);
  if (route) return route;  // Success
}
throw new RouteError('All models exhausted …', 429, diag);  // server/src/services/router.ts

```

When a model fails due to rate limiting, the router calls `recordRateLimitHit()` to apply a penalty that pushes the model down the priority list for subsequent requests. These penalties decay every **2 minutes**, allowing a model to regain its position once limits clear. If the entire chain is exhausted without finding a viable route, the system throws a `RouteError` with status code 429.

## Sticky Sessions and Pinned Routing

FreeLLMAPI supports session affinity through **pinned routing**. When a `preferredModelDbId` is provided (for example, from a previous chat turn), the router moves that specific model to the front of the chain using the `routePinnedModel` helper.

Unlike the standard fallback chain, pinned routing rotates only across the API keys of the specified model. If all keys for that model are exhausted, `routePinnedModel` returns `null` rather than falling back to a different model, allowing the caller to decide whether to trigger a full re-route.

```typescript
// Pin to model DB ID 42 from previous turn
const pinnedResult = routePinnedModel(42, 1200);
if (pinnedResult) {
  console.log('Pinned model served request');
} else {
  console.log('Pinned model unavailable');
}

```

## Implementation Examples

### Automatic Routing with Default Strategy

The `routeRequest` function encapsulates the full selection logic. It builds the active chain, applies the default `balanced` strategy, and returns the first viable model/key pair.

```typescript
import { routeRequest } from '@freellmapi/server';

// Estimate 1500 tokens for prompt + max completion
const result = routeRequest(1500);
console.log(`Using provider ${result.provider.name}, model ${result.modelId}`);

```

### Custom Weight Vectors

Users can override the scoring weights to favor specific characteristics. The `setCustomWeights` function persists the configuration to `CUSTOM_WEIGHTS_KEY` and applies it to future requests.

```typescript
import { setCustomWeights, routeRequest } from '@freellmapi/server';

// Prioritize speed over reliability: 0.2 reliability, 0.7 speed, 0.1 intelligence
setCustomWeights({ reliability: 0.2, speed: 0.7, intelligence: 0.1 });

const res = routeRequest(800);
console.log(`Chosen model: ${res.displayName}`);

```

## Summary

- **Dynamic chains**: The router fetches fresh configurations via `getActiveChain()` on every request to ensure current availability.
- **Intelligent scoring**: Multi-armed bandit algorithms in [`scoring.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/scoring.ts) rank models by reliability (Beta posterior), speed (latency/throughput), and intelligence tiers.
- **Automatic fallback**: A sequential loop in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) tries each model in ranked order, with 429 penalties decaying every 2 minutes.
- **Resource-aware selection**: `selectKeyForModel` validates rate limits, cooldowns, and token capacity before committing to a key.
- **Session pinning**: `routePinnedModel` isolates routing to a specific model’s keys without cross-model fallback.

## Frequently Asked Questions

### What routing strategies does FreeLLMAPI support?

FreeLLMAPI supports six strategies: **priority** (manual ordering), **balanced** (equal weighting), **smartest** (intelligence-weighted), **fastest** (latency-weighted), **reliable** (uptime-weighted), and **custom** (user-defined vector). Custom weights are persisted to `CUSTOM_WEIGHTS_KEY` and allow fine-tuning of the reliability, speed, and intelligence multipliers.

### How does the router handle rate limits and cooldowns?

The `selectKeyForModel` function checks per-minute and per-day limits via [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts). Keys exceeding these limits or in an active cooldown are skipped. When a rate limit is hit, `recordRateLimitHit()` applies a penalty score that decays over **2 minutes**, temporarily demoting the affected model in the fallback chain.

### What is the difference between automatic fallback and pinned routing?

**Automatic fallback** iterates through the entire sorted chain of models until any viable key is found. **Pinned routing** via `routePinnedModel` restricts selection to keys of a specific model (identified by `preferredModelDbId`) and returns `null` if that model is unavailable, preventing automatic fallback to alternative models.

### How often does the scoring penalty decay for rate-limited models?

Penalty weights applied after rate-limit hits decay every **2 minutes**. This allows the bandit scoring engine to automatically restore a model's priority once its quota resets or cooldown expires, ensuring the fallback chain reflects real-time capacity without manual intervention.