How FreeLLMAPI Embeddings Work and Why They Cannot Fail Over Across Different Models

FreeLLMAPI treats embeddings as a family-scoped service where fail-over is restricted to providers within the same vector space, preventing the corruption of downstream vector stores that would occur if incompatible embedding models were mixed.

FreeLLMAPI is an open-source LLM gateway that unifies multiple providers under a single API. Unlike chat completions, which can transparently fail over between different models, FreeLLMAPI embeddings operate under strict family-based constraints to ensure vector consistency. Understanding this architecture is critical for building reliable RAG pipelines and semantic search systems.

The Core Constraint: No Cross-Model Failover

The fundamental reason FreeLLMAPI embeddings cannot fail over across different models is that vectors from different models live in incompatible spaces. Silently switching models would corrupt any vector store built on top of the API, as downstream algorithms assume a consistent metric space for similarity search and clustering.

This rule is explicitly enforced at the top of the embeddings service in server/src/services/embeddings.ts:

// Embeddings routing. Unlike chat, embeddings can NOT fail over across models:
// vectors from different models live in incompatible spaces, and silently
// switching models would corrupt any vector store built on top of us.

Mixing vectors from two different families—such as OpenAI text-embedding-3-small and Google gemini-embedding-001—would produce meaningless distance calculations, rendering semantic search results useless.

Family-Based Routing Architecture

FreeLLMAPI implements a family-scoped routing system that groups providers serving the exact same vector space. This ensures that every embedding returned in a single request shares the same dimensionality and semantic encoding.

Resolving the Target Family

When a request arrives, the resolveFamily() function determines which vector space to use. The client can pass model: "auto" (or omit the field), which resolves to the default family (e.g., gemini-embedding-001), or specify a concrete model identifier.

In server/src/services/embeddings.ts, the resolution logic works as follows:

export function resolveFamily(model: string | undefined): string | null {
  if (!model || model === 'auto') return getDefaultFamily();          // ← default family
  const rows = listEmbeddingModels();
  if (rows.some(r => r.family === model)) return model;               // ← family name
  const byModelId = rows.find(r => r.model_id === model);
  return byModelId?.family ?? null;                                   // ← lookup by model id
}

If the provided string matches a family name directly, it returns that family; otherwise, it looks up the model ID to find its corresponding family.

Building the Provider Chain

Once the family is resolved, runEmbeddings() constructs a provider chain containing only enabled rows belonging to that specific family, ordered by priority:

const chain = getDb()
  .prepare('SELECT * FROM embedding_models WHERE family = ? AND enabled = 1 ORDER BY priority')
  .all(family) as EmbeddingModelRow[];

This SQL query ensures that fail-over candidates are strictly limited to providers within the same family, guaranteeing dimensional consistency.

Executing the Failover Loop

The service iterates over the provider chain, attempting each credential in sequence. If a provider returns an error, the next provider in the same family is tried. No provider from a different family will ever be considered:

for (const row of chain) {
  const credential = getProviderCredential(row);
  if (!credential) continue;                     // skip if no usable key
  try {
    const out = await callProvider(row, credential, inputs, dimensions);
    // ... processing ...
    return { /* EmbeddingsResult */ };
  } catch (err) {
    // remember the error then keep trying the next provider of the same family
  }
}

This implementation in server/src/services/embeddings.ts (lines 68-94) ensures that while individual providers may fail, the semantic integrity of the vector space remains intact.

Practical Implementation Examples

Using the HTTP API

The embeddings endpoint is exposed via embeddingsRouter in server/src/routes/embeddings.ts and accepts requests with either a specific family or "auto":

POST /api/embeddings HTTP/1.1
Host: your-freellmapi-instance.com
Authorization: Bearer <your-api-key>
Content-Type: application/json

{
  "model": "auto",
  "input": [
    "What is the capital of France?",
    "Explain quantum entanglement in plain language."
  ]
}

The response includes the specific family used and the resulting vectors:

{
  "object": "list",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.12, -0.05, ...] },
    { "object": "embedding", "index": 1, "embedding": [0.09, 0.33, ...] }
  ],
  "model": "gemini-embedding-001",
  "usage": { "prompt_tokens": 42 }
}

Direct Service Invocation

For backend integrations, import runEmbeddings() directly:

import { runEmbeddings } from './services/embeddings.js';

async function getEmbeddings() {
  const texts = [
    'Summarize the plot of "Hamlet".',
    'What is the boiling point of water?'
  ];

  // Resolves to default family (e.g., gemini-embedding-001)
  const result = await runEmbeddings('auto', texts);
  
  console.log('Family:', result.family);
  console.log('Dimensions:', result.dimensions);
  console.log('Vectors:', result.vectors);
}

The function returns an EmbeddingsResult object containing the family, platform, modelId, dimensions, vectors, and token usage, ensuring full traceability of which provider generated the embeddings.

Summary

• FreeLLMAPI embeddings are family-scoped: Unlike chat completions, embeddings cannot fail over across different models because vectors from different families occupy incompatible semantic spaces.

• Family resolution is strict: The resolveFamily() function in server/src/services/embeddings.ts maps requests to specific vector spaces, supporting "auto" resolution or explicit family/model IDs.

• Failover is intra-family only: The provider chain built by runEmbeddings() queries embedding_models filtered by family, ensuring all fallback providers share identical dimensionality.

• Vector store integrity is protected: By preventing cross-model failover, FreeLLMAPI guarantees that all embeddings in a request share the same metric space, preserving the accuracy of similarity search and RAG pipelines.

Frequently Asked Questions

What happens if I request "model": "auto" for embeddings?

When you specify "model": "auto" or omit the model field entirely, FreeLLMAPI calls getDefaultFamily() to resolve the request to the configured default embedding family (typically gemini-embedding-001). This ensures consistent vector generation while allowing administrators to switch default providers globally without changing client code.

Can I force FreeLLMAPI to use a specific embedding provider?

Yes, you can pass a specific model ID (e.g., "text-embedding-3-small") or family name (e.g., "gemini-embedding-001") in the request. The resolveFamily() function will validate this against the embedding_models table and restrict failover to other providers within that same family only.

Why do embedding vectors from different models produce incompatible results?

Different embedding models are trained on distinct architectures and datasets, resulting in different vector spaces with unique dimensionalities and semantic mappings. A vector from OpenAI's text-embedding-3-small represents concepts in a fundamentally different coordinate system than Google's gemini-embedding-001. Mixing these in the same vector store would cause similarity calculations to return random distances, corrupting search results.

How does FreeLLMAPI handle provider errors within the same family?

If a provider in the chain returns an error or lacks valid credentials, runEmbeddings() catches the exception and continues to the next provider in the priority-ordered chain. This intra-family failover mechanism ensures high availability while maintaining the strict constraint that all attempted providers belong to the same vector space, as enforced by the SQL query in server/src/services/embeddings.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}