Embeddings Routing vs Chat Routing in FreeLLMAPI: Architectural Differences Explained

Chat routing uses an adaptive multi-armed bandit algorithm to score and switch between any enabled models, while embeddings routing binds requests to a specific family and only falls back within that same family to guarantee vector space consistency.

FreeLLMAPI implements two distinct request routing subsystems optimized for their respective workloads. Understanding how embeddings routing differs from chat routing is critical for architects building reliable AI applications, as the wrong assumption about fallback behavior can corrupt vector databases or introduce unpredictable model switching in conversational flows.

Entry Points and Request Handling

The two subsystems begin at completely different entry points in the codebase.

Chat routing initiates through the routeRequest() function in server/src/services/router.ts. This function handles core LLM calls and accepts parameters like estimated token counts, vision requirements, and tool requirements.

Embeddings routing flows through runEmbeddings() in server/src/services/embeddings.ts. This function resolves the request to a specific embedding family (e.g., text-embedding-004) before attempting any provider calls.

Model Selection Strategy

The fundamental difference lies in how each system selects which model to use.

Chat routing offers flexible selection. The system can fall back to any enabled model across different providers, even switching between models with different dimensions or tokenizers. Selection is driven by a configurable routing strategy (priority, balanced, smartest, etc.) that scores every available model using a multi-armed bandit algorithm.

Embeddings routing is rigidly family-bound. Each request must resolve to a specific family, and only providers serving that exact family are considered. The system never performs cross-family fallbacks because different embedding families produce incompatible vector spaces. Attempting to mix text-embedding-004 vectors with text-embedding-3-small vectors would corrupt a vector store.

Scoring and Prioritization Logic

The algorithms used to rank available models differ significantly between the two systems.

Chat routing implements sophisticated scoring in the orderChain function (lines 69-88 of router.ts). It calculates bandit scores using reliabilityPosterior, speedScore, and intelligenceScore, then applies Thompson sampling for exploration. Custom weights stored in routing_custom_weights further refine the ordering, and ties are broken by the manual priority column.

Embeddings routing uses a simple static priority. After resolving the family via resolveFamily (lines 54-62 of embeddings.ts), the system queries the embedding_models table with ORDER BY priority. No bandit logic, no custom weights, and no sampling occur—only the static priority column determines provider order.

API Key Management

Key rotation strategies reflect the different resiliency requirements of each workload.

Chat routing implements per-model round-robin across all enabled API keys. The selectKeyForModel function (lines 46-100 of router.ts) checks rate limits, cooldowns, and quotas for each key. When a provider returns a 429 response, the system records the hit via recordRateLimitHit, potentially demoting that model in future scoring.

Embeddings routing looks up one usable key per provider using getPlatformKey. Since each family typically has at most one key per provider, no round-robin occurs. If a provider fails, the system simply moves to the next provider in the same family.

Failure Handling and Fallback Behavior

Failure recovery patterns represent the most critical architectural distinction.

Chat routing walks the entire orderChain, potentially switching to a completely different model with different capabilities if the primary choice fails. This is acceptable for conversational workloads where returning a response is prioritized over maintaining model consistency.

Embeddings routing strictly limits fallbacks to providers within the same family. If all providers for the requested family fail, the request aborts. The system never falls back to a different family, ensuring that all returned vectors share identical dimensions and semantic spaces. As implemented in runEmbeddings (lines 202-214 of embeddings.ts), the loop iterates only through providers matching the resolved family.

Configuration and Persistence

Both systems store their configuration in the database, but use different settings tables.

Chat routing persists the active strategy in the routing_strategy setting, read via getRoutingStrategy (lines 54-59 of router.ts). Users can change strategies at runtime (priority, balanced, smartest) and customize weights via routing_custom_weights.

Embeddings routing uses the embeddings_default_family setting, accessed through getDefaultFamily (lines 50-52 of embeddings.ts). When users pass undefined as the family parameter, the system defaults to this configured value.

Logging and Observability

Request tracking differs to reflect the distinct optimization needs.

Chat routing logs every request to the requests table, capturing model-level statistics that feed back into the bandit algorithm for continuous scoring refinement.

Embeddings routing also logs to the requests table but sets request_type = 'embedding'. However, only provider-level data matters for basic reliability tracking; no per-model statistics feed back into the routing logic since the priority-based system does not adapt based on historical performance.

Implementation in Code

Chat Request with Adaptive Routing

import { routeRequest } from './router';

// Request chat completion with 1500 estimated tokens
const result = routeRequest(
  1500,                 // estimatedTokens
  undefined,           // skipKeys
  undefined,           // preferredModelDbId
  false,               // requireVision
  false                // requireTools
);

console.log(`Provider: ${result.provider.name}, Model: ${result.modelId}`);

This call triggers the full bandit scoring system, potentially selecting different models for each request based on recent performance data.

Embedding Request with Family Binding

import { runEmbeddings } from './embeddings';

const inputs = [
  "The quick brown fox jumps over the lazy dog.",
  "FreeLLMAPI provides a unified API for many LLM providers."
];

// Undefined uses the default family from settings
const embeddings = await runEmbeddings(undefined, inputs);
console.log(`Family ${embeddings.family} returned ${embeddings.vectors.length} vectors`);

This call resolves to the default family (via getDefaultFamily), then attempts providers in strict priority order within that family only.

Summary

• Chat routing in server/src/services/router.ts uses the routeRequest() function with adaptive bandit scoring to select from any enabled model, while embeddings routing in server/src/services/embeddings.ts uses runEmbeddings() with strict family binding.
• Chat routing implements Thompson sampling and custom weights in orderChain (lines 69-88), while embeddings routing uses simple ORDER BY priority queries.
• Chat routing rotates through multiple keys per model via selectKeyForModel (lines 46-100), while embeddings routing uses a single key per provider via getPlatformKey.
• Failure handling in chat routing can switch to any model, but embeddings routing aborts if all providers in the target family fail, never crossing family boundaries.
• Configuration persists via routing_strategy for chat and embeddings_default_family for embeddings.

Frequently Asked Questions

Can FreeLLMAPI fall back to a different embedding family if the primary family fails?

No. According to the source code in server/src/services/embeddings.ts (lines 202-214), the runEmbeddings function strictly iterates through providers within the resolved family. If all providers for that family fail, the request aborts. This prevents vector space incompatibility that would occur if the system switched from text-embedding-004 to a different dimension family mid-request.

Why does chat routing use a multi-armed bandit algorithm while embeddings routing does not?

Chat routing optimizes for conversational quality and availability across diverse models with varying capabilities, requiring adaptive scoring based on reliabilityPosterior, speedScore, and intelligenceScore. Embeddings routing optimizes for vector consistency, where the mathematical properties of the embedding space matter more than marginal performance differences. Since mixing vector spaces corrupts retrieval accuracy, the system uses static priority ordering instead of adaptive bandit scoring.

How does the system handle API key exhaustion differently between chat and embeddings?

Chat routing implements sophisticated round-robin through selectKeyForModel in router.ts, tracking rate limits, cooldowns, and quotas across multiple keys per model, with 429 penalties affecting future scoring. Embeddings routing simply looks up one usable key per provider via getPlatformKey and retries within the same family, assuming at most one key per provider per family.

Where are the routing strategies stored in FreeLLMAPI?

Chat routing strategies persist in the routing_strategy setting, accessed via getRoutingStrategy (lines 54-59 of router.ts), while the default embedding family stores in embeddings_default_family, accessed via getDefaultFamily (lines 50-52 of embeddings.ts). Both use the database settings layer defined in server/src/db/index.js.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}