# Embeddings Routing vs Chat Routing in FreeLLMAPI: Architectural Differences Explained

> Understand the core architectural differences between embeddings routing and chat routing in FreeLLMAPI. Learn how each method handles model selection and consistency for optimal performance.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: architecture
- Published: 2026-06-24

---

**Chat routing uses an adaptive multi-armed bandit algorithm to score and switch between any enabled models, while embeddings routing binds requests to a specific family and only falls back within that same family to guarantee vector space consistency.**

FreeLLMAPI implements two distinct request routing subsystems optimized for their respective workloads. Understanding how **embeddings routing** differs from **chat routing** is critical for architects building reliable AI applications, as the wrong assumption about fallback behavior can corrupt vector databases or introduce unpredictable model switching in conversational flows.

## Entry Points and Request Handling

The two subsystems begin at completely different entry points in the codebase.

**Chat routing** initiates through the `routeRequest()` function in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts). This function handles core LLM calls and accepts parameters like estimated token counts, vision requirements, and tool requirements.

**Embeddings routing** flows through `runEmbeddings()` in [`server/src/services/embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/embeddings.ts). This function resolves the request to a specific embedding family (e.g., `text-embedding-004`) before attempting any provider calls.

## Model Selection Strategy

The fundamental difference lies in how each system selects which model to use.

**Chat routing** offers flexible selection. The system can fall back to any enabled model across different providers, even switching between models with different dimensions or tokenizers. Selection is driven by a configurable routing strategy (priority, balanced, smartest, etc.) that scores every available model using a multi-armed bandit algorithm.

**Embeddings routing** is rigidly family-bound. Each request must resolve to a specific family, and only providers serving that exact family are considered. The system never performs cross-family fallbacks because different embedding families produce incompatible vector spaces. Attempting to mix `text-embedding-004` vectors with `text-embedding-3-small` vectors would corrupt a vector store.

## Scoring and Prioritization Logic

The algorithms used to rank available models differ significantly between the two systems.

**Chat routing** implements sophisticated scoring in the `orderChain` function (lines 69-88 of [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)). It calculates bandit scores using `reliabilityPosterior`, `speedScore`, and `intelligenceScore`, then applies Thompson sampling for exploration. Custom weights stored in `routing_custom_weights` further refine the ordering, and ties are broken by the manual `priority` column.

**Embeddings routing** uses a simple static priority. After resolving the family via `resolveFamily` (lines 54-62 of [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)), the system queries the `embedding_models` table with `ORDER BY priority`. No bandit logic, no custom weights, and no sampling occur—only the static `priority` column determines provider order.

## API Key Management

Key rotation strategies reflect the different resiliency requirements of each workload.

**Chat routing** implements per-model round-robin across all enabled API keys. The `selectKeyForModel` function (lines 46-100 of [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)) checks rate limits, cooldowns, and quotas for each key. When a provider returns a 429 response, the system records the hit via `recordRateLimitHit`, potentially demoting that model in future scoring.

**Embeddings routing** looks up one usable key per provider using `getPlatformKey`. Since each family typically has at most one key per provider, no round-robin occurs. If a provider fails, the system simply moves to the next provider in the same family.

## Failure Handling and Fallback Behavior

Failure recovery patterns represent the most critical architectural distinction.

**Chat routing** walks the entire `orderChain`, potentially switching to a completely different model with different capabilities if the primary choice fails. This is acceptable for conversational workloads where returning a response is prioritized over maintaining model consistency.

**Embeddings routing** strictly limits fallbacks to providers within the same family. If all providers for the requested family fail, the request aborts. The system never falls back to a different family, ensuring that all returned vectors share identical dimensions and semantic spaces. As implemented in `runEmbeddings` (lines 202-214 of [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)), the loop iterates only through providers matching the resolved family.

## Configuration and Persistence

Both systems store their configuration in the database, but use different settings tables.

**Chat routing** persists the active strategy in the `routing_strategy` setting, read via `getRoutingStrategy` (lines 54-59 of [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)). Users can change strategies at runtime (priority, balanced, smartest) and customize weights via `routing_custom_weights`.

**Embeddings routing** uses the `embeddings_default_family` setting, accessed through `getDefaultFamily` (lines 50-52 of [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)). When users pass `undefined` as the family parameter, the system defaults to this configured value.

## Logging and Observability

Request tracking differs to reflect the distinct optimization needs.

**Chat routing** logs every request to the `requests` table, capturing model-level statistics that feed back into the bandit algorithm for continuous scoring refinement.

**Embeddings routing** also logs to the `requests` table but sets `request_type = 'embedding'`. However, only provider-level data matters for basic reliability tracking; no per-model statistics feed back into the routing logic since the priority-based system does not adapt based on historical performance.

## Implementation in Code

### Chat Request with Adaptive Routing

```typescript
import { routeRequest } from './router';

// Request chat completion with 1500 estimated tokens
const result = routeRequest(
  1500,                 // estimatedTokens
  undefined,           // skipKeys
  undefined,           // preferredModelDbId
  false,               // requireVision
  false                // requireTools
);

console.log(`Provider: ${result.provider.name}, Model: ${result.modelId}`);

```

This call triggers the full bandit scoring system, potentially selecting different models for each request based on recent performance data.

### Embedding Request with Family Binding

```typescript
import { runEmbeddings } from './embeddings';

const inputs = [
  "The quick brown fox jumps over the lazy dog.",
  "FreeLLMAPI provides a unified API for many LLM providers."
];

// Undefined uses the default family from settings
const embeddings = await runEmbeddings(undefined, inputs);
console.log(`Family ${embeddings.family} returned ${embeddings.vectors.length} vectors`);

```

This call resolves to the default family (via `getDefaultFamily`), then attempts providers in strict priority order within that family only.

## Summary

- **Chat routing** in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) uses the `routeRequest()` function with adaptive bandit scoring to select from any enabled model, while **embeddings routing** in [`server/src/services/embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/embeddings.ts) uses `runEmbeddings()` with strict family binding.
- Chat routing implements Thompson sampling and custom weights in `orderChain` (lines 69-88), while embeddings routing uses simple `ORDER BY priority` queries.
- Chat routing rotates through multiple keys per model via `selectKeyForModel` (lines 46-100), while embeddings routing uses a single key per provider via `getPlatformKey`.
- Failure handling in chat routing can switch to any model, but embeddings routing aborts if all providers in the target family fail, never crossing family boundaries.
- Configuration persists via `routing_strategy` for chat and `embeddings_default_family` for embeddings.

## Frequently Asked Questions

### Can FreeLLMAPI fall back to a different embedding family if the primary family fails?

No. According to the source code in [`server/src/services/embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/embeddings.ts) (lines 202-214), the `runEmbeddings` function strictly iterates through providers within the resolved family. If all providers for that family fail, the request aborts. This prevents vector space incompatibility that would occur if the system switched from `text-embedding-004` to a different dimension family mid-request.

### Why does chat routing use a multi-armed bandit algorithm while embeddings routing does not?

Chat routing optimizes for conversational quality and availability across diverse models with varying capabilities, requiring adaptive scoring based on `reliabilityPosterior`, `speedScore`, and `intelligenceScore`. Embeddings routing optimizes for vector consistency, where the mathematical properties of the embedding space matter more than marginal performance differences. Since mixing vector spaces corrupts retrieval accuracy, the system uses static `priority` ordering instead of adaptive bandit scoring.

### How does the system handle API key exhaustion differently between chat and embeddings?

Chat routing implements sophisticated round-robin through `selectKeyForModel` in [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts), tracking rate limits, cooldowns, and quotas across multiple keys per model, with 429 penalties affecting future scoring. Embeddings routing simply looks up one usable key per provider via `getPlatformKey` and retries within the same family, assuming at most one key per provider per family.

### Where are the routing strategies stored in FreeLLMAPI?

Chat routing strategies persist in the `routing_strategy` setting, accessed via `getRoutingStrategy` (lines 54-59 of [`router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/router.ts)), while the default embedding family stores in `embeddings_default_family`, accessed via `getDefaultFamily` (lines 50-52 of [`embeddings.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/embeddings.ts)). Both use the database settings layer defined in [`server/src/db/index.js`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/db/index.js).