How Embeddings Routing Differs from Chat Routing in FreeLLMAPI

FreeLLMAPI implements two distinct routing architectures: chat routing uses an adaptive multi-armed bandit algorithm to switch between any enabled models, while embeddings routing is restricted to a specific model family (e.g., text-embedding-004) to ensure vector space consistency.

FreeLLMAPI is an open-source unified API for large language models that separates conversational workloads from vector generation tasks. Understanding how embeddings routing differs from chat routing is critical for optimizing both response quality and retrieval accuracy. This guide examines the source code implementation in server/src/services/router.ts and server/src/services/embeddings.ts to reveal the architectural distinctions.

Entry Points and Model Selection

The two routing systems begin at entirely different entry points with opposite flexibility constraints.

Chat routing enters through routeRequest() in router.ts and can flexibly fall back to any enabled model across providers. Selection is driven by a configurable routing strategy that scores models dynamically, allowing the system to switch between different architectures (e.g., from GPT-4 to Claude) mid-request if the primary choice fails.

Embeddings routing enters through runEmbeddings() in embeddings.ts and binds each request to a specific family (e.g., text-embedding-004). Only providers serving that exact family are considered. Because vector spaces are incompatible across different embedding families, the system never attempts cross-family fallback, preserving the mathematical consistency required for vector search.

Scoring and Ordering Logic

The methods for ordering candidate providers differ fundamentally between the two systems.

Chat routing employs a multi-armed bandit algorithm with Thompson sampling. The orderChain function (lines 69-88 in router.ts) computes scores using reliabilityPosterior, speedScore, and intelligenceScore combined with custom weights. These scores are re-sampled on each request to balance exploration and exploitation, producing a dynamic ranking of all available models.

Embeddings routing uses a static priority system. The runEmbeddings function queries the embedding_models table with a simple ORDER BY priority clause. No bandit logic, custom weights, or intelligence scoring is applied—only the static priority column determines the provider order.

Key Rotation and Failure Handling

Failure recovery mechanisms reflect the different consistency requirements of each workload.

Chat routing rotates through all enabled API keys per model via selectKeyForModel (lines 46-100 in router.ts). The system applies rate-limit checks, cooldown periods, and quota validations. When a provider returns a 429 error, recordRateLimitHit penalizes the model's bandit score, potentially demoting it in future requests. If a specific key fails, the router walks the entire orderChain, possibly switching to a completely different model with different tokenization or context windows.

Embeddings routing retrieves one usable key per provider via getPlatformKey and retries within the same family only. The loop (lines 202-214 in embeddings.ts) attempts each provider sequentially; if one fails, it proceeds to the next provider in the same family. If all providers for that family fail, the request aborts. There is no round-robin across multiple keys per provider because the family is expected to have at most one key per platform.

Configuration and Persistence

Both systems store their configuration in the database settings table but use different keys and schemas.

Chat routing strategy is retrieved via getRoutingStrategy (lines 54-59 in router.ts) using the setting key routing_strategy. This supports runtime changes and persists custom weights in routing_custom_weights.

Embeddings routing uses the embeddings_default_family setting accessed through getDefaultFamily (lines 50-52 in embeddings.ts). Individual families are defined in the embedding_models table with a static priority column, while chat routing logs every request to the requests table with model-level statistics used for bandit feedback. Embeddings requests are also logged to the requests table with request_type = 'embedding', but provider-level data is not fed back into routing decisions.

Code Implementation Examples

Chat Routing with Dynamic Model Selection

import { routeRequest } from './router';

// Route a chat completion with estimated token count
const result = routeRequest(
  1500,                 // estimatedTokens
  undefined,           // skipKeys
  undefined,           // preferredModelDbId (no sticky model)
  false,               // requireVision
  false                // requireTools
);

console.log(`Selected: ${result.provider.name} with model ${result.modelId}`);

The routeRequest function reads the current routing_strategy, invokes orderChain to score all enabled models using bandit weights, and iterates through the chain. If a specific key hits a rate limit, the system may switch to an entirely different model architecture to fulfill the request.

Embeddings Routing with Family Lock

import { runEmbeddings } from './embeddings';

const inputs = [
  "Vector consistency requires the same embedding space.",
  "FreeLLMAPI enforces this through family-bound routing."
];

// undefined selects the default family from settings
const result = await runEmbeddings(undefined, inputs);
console.log(`Family ${result.family} returned ${result.vectors.length} vectors`);

The runEmbeddings function calls resolveFamily (lines 54-62 in embeddings.ts) to determine the target family, then queries the embedding_models table for providers in that family ordered by priority. The system guarantees that all returned vectors share the same dimensionality by never falling back to a different family.

Summary

• Chat routing uses adaptive multi-armed bandit scoring via orderChain in router.ts to dynamically select models across providers, supporting cross-model fallback and key rotation.
• Embeddings routing restricts requests to a specific model family via resolveFamily in embeddings.ts, using static priority-based ordering and limiting fallback to providers within the same vector space.
• Key management differs: chat routing round-robins through all keys per model with rate-limit penalties, while embeddings routing selects one key per provider without rotation.
• Vector consistency is protected by preventing cross-family fallback in embeddings, whereas chat routing prioritizes availability over model consistency.

Frequently Asked Questions

Can embeddings routing fall back to a different model family if the primary family fails?

No. According to the source code in server/src/services/embeddings.ts, the runEmbeddings function iterates only through providers sharing the same family (lines 202-214). If all providers in that family fail, the request aborts. This restriction exists because different embedding families produce incompatible vector spaces that would corrupt retrieval results.

How does chat routing decide which model to use when multiple providers are available?

Chat routing employs a multi-armed bandit algorithm implemented in the orderChain function (router.ts lines 69-88). It calculates scores using reliabilityPosterior, speedScore, and intelligenceScore with custom weights, then applies Thompson sampling to balance exploration and exploitation. The router then attempts models in this scored order, rotating through API keys via selectKeyForModel.

What happens when a chat provider returns a 429 rate limit error?

The selectKeyForModel function (lines 46-100 in router.ts) records the rate limit hit via recordRateLimitHit, which applies a penalty to the model's bandit score. This demotes the model in future routing decisions. The router immediately attempts the next model in the orderChain, which may be a completely different model from a different provider.

Where are the routing strategies and default embedding families stored?

Both are stored in the database settings table. Chat routing strategy is retrieved via getRoutingStrategy (lines 54-59 in router.ts) using the key routing_strategy. Embeddings default family is retrieved via getDefaultFamily (lines 50-52 in embeddings.ts) using the key embeddings_default_family. Individual embedding families are configured in the embedding_models table with a priority column.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}