# How Embeddings Routing Differs from Chat Routing in FreeLLMAPI

> Discover how FreeLLMAPI's embeddings routing differs from chat routing. Learn about adaptive multi-armed bandits versus specific model families and vector space consistency.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: deep-dive
- Published: 2026-06-28

---

**FreeLLMAPI implements two distinct routing architectures: chat routing uses an adaptive multi-armed bandit algorithm to switch between any enabled models, while embeddings routing is restricted to a specific model family (e.g., `text-embedding-004`) to ensure vector space consistency.**

FreeLLMAPI is an open-source unified API for large language models that separates conversational workloads from vector generation tasks. Understanding how embeddings routing differs from chat routing is critical for optimizing both response quality and retrieval accuracy. This guide examines the source code implementation in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) and [`server/src/services/embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/embeddings.ts) to reveal the architectural distinctions.

## Entry Points and Model Selection

The two routing systems begin at entirely different entry points with opposite flexibility constraints.

**Chat routing** enters through `routeRequest()` in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) and can flexibly fall back to any enabled model across providers. Selection is driven by a configurable routing strategy that scores models dynamically, allowing the system to switch between different architectures (e.g., from GPT-4 to Claude) mid-request if the primary choice fails.

**Embeddings routing** enters through `runEmbeddings()` in [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts) and binds each request to a specific *family* (e.g., `text-embedding-004`). Only providers serving that exact family are considered. Because vector spaces are incompatible across different embedding families, the system never attempts cross-family fallback, preserving the mathematical consistency required for vector search.

## Scoring and Ordering Logic

The methods for ordering candidate providers differ fundamentally between the two systems.

**Chat routing** employs a multi-armed bandit algorithm with Thompson sampling. The `orderChain` function (lines 69-88 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) computes scores using `reliabilityPosterior`, `speedScore`, and `intelligenceScore` combined with custom weights. These scores are re-sampled on each request to balance exploration and exploitation, producing a dynamic ranking of all available models.

**Embeddings routing** uses a static priority system. The `runEmbeddings` function queries the `embedding_models` table with a simple `ORDER BY priority` clause. No bandit logic, custom weights, or intelligence scoring is applied—only the static `priority` column determines the provider order.

## Key Rotation and Failure Handling

Failure recovery mechanisms reflect the different consistency requirements of each workload.

**Chat routing** rotates through all enabled API keys per model via `selectKeyForModel` (lines 46-100 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)). The system applies rate-limit checks, cooldown periods, and quota validations. When a provider returns a 429 error, `recordRateLimitHit` penalizes the model's bandit score, potentially demoting it in future requests. If a specific key fails, the router walks the entire `orderChain`, possibly switching to a completely different model with different tokenization or context windows.

**Embeddings routing** retrieves one usable key per provider via `getPlatformKey` and retries within the same family only. The loop (lines 202-214 in [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)) attempts each provider sequentially; if one fails, it proceeds to the next provider in the same family. If all providers for that family fail, the request aborts. There is no round-robin across multiple keys per provider because the family is expected to have at most one key per platform.

## Configuration and Persistence

Both systems store their configuration in the database settings table but use different keys and schemas.

**Chat routing** strategy is retrieved via `getRoutingStrategy` (lines 54-59 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) using the setting key `routing_strategy`. This supports runtime changes and persists custom weights in `routing_custom_weights`.

**Embeddings routing** uses the `embeddings_default_family` setting accessed through `getDefaultFamily` (lines 50-52 in [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)). Individual families are defined in the `embedding_models` table with a static `priority` column, while chat routing logs every request to the `requests` table with model-level statistics used for bandit feedback. Embeddings requests are also logged to the `requests` table with `request_type = 'embedding'`, but provider-level data is not fed back into routing decisions.

## Code Implementation Examples

### Chat Routing with Dynamic Model Selection

```typescript
import { routeRequest } from './router';

// Route a chat completion with estimated token count
const result = routeRequest(
  1500,                 // estimatedTokens
  undefined,           // skipKeys
  undefined,           // preferredModelDbId (no sticky model)
  false,               // requireVision
  false                // requireTools
);

console.log(`Selected: ${result.provider.name} with model ${result.modelId}`);

```

The `routeRequest` function reads the current `routing_strategy`, invokes `orderChain` to score all enabled models using bandit weights, and iterates through the chain. If a specific key hits a rate limit, the system may switch to an entirely different model architecture to fulfill the request.

### Embeddings Routing with Family Lock

```typescript
import { runEmbeddings } from './embeddings';

const inputs = [
  "Vector consistency requires the same embedding space.",
  "FreeLLMAPI enforces this through family-bound routing."
];

// undefined selects the default family from settings
const result = await runEmbeddings(undefined, inputs);
console.log(`Family ${result.family} returned ${result.vectors.length} vectors`);

```

The `runEmbeddings` function calls `resolveFamily` (lines 54-62 in [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)) to determine the target family, then queries the `embedding_models` table for providers in that family ordered by priority. The system guarantees that all returned vectors share the same dimensionality by never falling back to a different family.

## Summary

- **Chat routing** uses adaptive multi-armed bandit scoring via `orderChain` in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) to dynamically select models across providers, supporting cross-model fallback and key rotation.
- **Embeddings routing** restricts requests to a specific model family via `resolveFamily` in [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts), using static priority-based ordering and limiting fallback to providers within the same vector space.
- **Key management** differs: chat routing round-robins through all keys per model with rate-limit penalties, while embeddings routing selects one key per provider without rotation.
- **Vector consistency** is protected by preventing cross-family fallback in embeddings, whereas chat routing prioritizes availability over model consistency.

## Frequently Asked Questions

### Can embeddings routing fall back to a different model family if the primary family fails?

No. According to the source code in [`server/src/services/embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/embeddings.ts), the `runEmbeddings` function iterates only through providers sharing the same family (lines 202-214). If all providers in that family fail, the request aborts. This restriction exists because different embedding families produce incompatible vector spaces that would corrupt retrieval results.

### How does chat routing decide which model to use when multiple providers are available?

Chat routing employs a multi-armed bandit algorithm implemented in the `orderChain` function ([`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts) lines 69-88). It calculates scores using `reliabilityPosterior`, `speedScore`, and `intelligenceScore` with custom weights, then applies Thompson sampling to balance exploration and exploitation. The router then attempts models in this scored order, rotating through API keys via `selectKeyForModel`.

### What happens when a chat provider returns a 429 rate limit error?

The `selectKeyForModel` function (lines 46-100 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) records the rate limit hit via `recordRateLimitHit`, which applies a penalty to the model's bandit score. This demotes the model in future routing decisions. The router immediately attempts the next model in the `orderChain`, which may be a completely different model from a different provider.

### Where are the routing strategies and default embedding families stored?

Both are stored in the database settings table. Chat routing strategy is retrieved via `getRoutingStrategy` (lines 54-59 in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) using the key `routing_strategy`. Embeddings default family is retrieved via `getDefaultFamily` (lines 50-52 in [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)) using the key `embeddings_default_family`. Individual embedding families are configured in the `embedding_models` table with a `priority` column.