Data Flow for a Client Request in FreeLLMAPI: End-to-End Architecture

A client request in FreeLLMAPI flows through Express middleware, unified API key authentication, Zod validation, intelligent model routing with sticky sessions, key selection with rate-limit checks, and provider execution with streaming response handling.

The data flow for a client request in FreeLLMAPI orchestrates a sophisticated pipeline across the tashfeenahmed/freellmapi repository. Every request enters through server/src/app.ts and traverses the proxy layer in server/src/routes/proxy.ts before reaching the intelligent router in server/src/services/router.ts. This architecture enables unified API key management, automatic failover between LLM providers, and OpenAI-compatible endpoints while maintaining session continuity.

Phase 1: Request Ingestion and Authentication

Express Application Bootstrap

In server/src/app.ts, the Express application initializes with security middleware including helmet and cors. The server mounts all API routes under /api/* for dashboard operations and /v1/* for the OpenAI-compatible proxy interface.

Unified API Key Extraction

Every request hitting the /v1/* proxy endpoints passes through extractApiToken in server/src/routes/proxy.ts. This function accepts either Authorization: Bearer ... headers or x-api-key headers, then performs a timing-safe equality check via timingSafeStringEqual to validate tokens against the stored unified key.

Phase 2: Validation and Model Resolution

Zod Schema Validation

Before routing logic executes, the request body undergoes strict validation using Zod schemas defined in server/src/routes/proxy.ts. The chatCompletionSchema validates message structure, temperature settings, and tool configurations. Invalid payloads receive an immediate 400 error response before any provider logic runs.

Model Selection Strategy

The routeRequest function in server/src/services/router.ts determines the target model through resolveRoutingChain. The algorithm:

• Resolves the auto-routing chain for fallback options
• Applies the configured routing strategy (priority, balanced, or smartest) using scoreChainEntry
• Filters models by capabilities (vision, tools) and context-window limits
• Respects sticky sessions via getStickyModel and setStickyModel to maintain conversation continuity across requests

Phase 3: Key Selection and Context Handoff

API Key Management

For the selected model, selectKeyForModel in server/src/services/router.ts implements round-robin selection per platform while checking cooldowns, provider health, and quota limits. If a key fails, the router records the failure via recordRateLimitHit and attempts the next model in the chain.

Session Continuity

When auto-routing switches models mid-conversation, maybeInjectContextHandoff in server/src/services/context-handoff.ts prepends a system message keyed by getSessionKey. This informs the new model that it is taking over an existing conversation, preserving context across provider switches.

Phase 4: Provider Execution and Response

Downstream Provider Invocation

The router returns a RouteResult object containing provider, modelId, and the decrypted apiKey. The proxy then invokes provider.streamChatCompletion (or the non-streaming equivalent) with the prepared messages, temperature, and tool settings.

Streaming and Tool Call Handling

For streaming requests, server/src/routes/proxy.ts opens an SSE connection with Content-Type: text/event-stream, buffers tool-call deltas, rescues inline-dialect tool markers, and ensures every turn ends with a proper finish_reason. Errors encountered mid-stream convert to JSON error frames rather than crashing the connection.

Response Finalization

After provider completion, the server adds usage statistics, sets headers (X-Routed-Via, X-Request-ID), writes the final [DONE] frame for SSE, or returns a JSON payload for non-streaming calls via res.json(response).

Phase 5: Analytics and Feedback Loops

Request Logging

Every routing attempt triggers traceRouteEvent from server/src/lib/request-log.ts, recording request ID, attempt number, latency, token counts, and any errors. Successful completions call recordSuccess while rate limits trigger recordRateLimitHit in server/src/services/router.ts, updating bandit scores that inform future routing decisions through the scoring algorithms in server/src/services/scoring.ts.

Code Examples: Consuming the FreeLLMAPI Proxy

Simple Chat Completion (cURL)

curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_UNIFIED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "auto",
        "messages": [{ "role": "user", "content": "Explain quantum entanglement in one sentence." }],
        "temperature": 0.7,
        "stream": false
      }'

This request hits proxyRouter.post('/chat/completions'), passes through authentication and validation, triggers the auto-router logic, and returns a JSON answer.

Streaming Response (Node.js)

import fetch from 'node-fetch';

const resp = await fetch('http://localhost:3000/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer YOUR_UNIFIED_API_KEY',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'auto',
    messages: [{ role: 'user', content: 'Write a short poem about sunrise.' }],
    stream: true,
  }),
});

const reader = resp.body.getReader();
let done = false;
while (!done) {
  const { value, done: end } = await reader.read();
  done = end;
  if (value) console.log(Buffer.from(value).toString()); // SSE chunks
}

The server opens an SSE stream, buffers tool-call deltas, and yields chunks until [DONE].

Pinned Model Request

curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_UNIFIED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [{"role":"user","content":"Summarize the plot of Inception."}],
        "stream": false
      }'

Because a concrete model ID is supplied, the proxy bypasses the auto-router and calls routePinnedModel via routeRequest with preferredModelDbId, ignoring sticky-session state.

Summary

• Request entry occurs through Express in server/src/app.ts, mounting routes under /v1/* for OpenAI-compatible endpoints.
• Authentication uses extractApiToken with timing-safe equality checks in server/src/routes/proxy.ts.
• Validation occurs via Zod schemas (chatCompletionSchema) before routing logic executes.
• Model selection happens in server/src/services/router.ts through resolveRoutingChain, supporting strategies like priority, balanced, and smartest with sticky session support.
• Key selection uses selectKeyForModel with round-robin logic, cooldown checks, and quota enforcement.
• Context handoff via maybeInjectContextHandoff preserves conversation state when switching models.
• Streaming execution handles SSE responses, tool-call rescue, and error frame injection in server/src/routes/proxy.ts.
• Analytics update bandit scores through traceRouteEvent, recordSuccess, and recordRateLimitHit to optimize future routing decisions.

Frequently Asked Questions

How does FreeLLMAPI handle authentication for incoming requests?

FreeLLMAPI extracts API keys through extractApiToken in server/src/routes/proxy.ts, which accepts both Authorization: Bearer headers and x-api-key headers. The system validates tokens using timingSafeStringEqual to prevent timing attacks against the unified key stored in the system configuration.

What happens when the requested model is unavailable or rate-limited?

The routeRequest function in server/src/services/router.ts implements a fallback chain through resolveRoutingChain. If selectKeyForModel encounters a rate-limited key, it calls recordRateLimitHit to update the model's bandit score and automatically attempts the next model in the chain until finding a healthy provider with available quota.

How does FreeLLMAPI maintain conversation context across different models?

When auto-routing switches models mid-conversation, the maybeInjectContextHandoff function in server/src/services/context-handoff.ts prepends a system message to the conversation. This message, keyed by getSessionKey, informs the new model that it is taking over from a previous assistant, preserving context continuity across provider switches.

What routing strategies are available for model selection?

FreeLLMAPI supports multiple routing strategies including priority, balanced, and smartest, implemented through scoreChainEntry in server/src/services/router.ts. The priority strategy favors specific providers, balanced distributes load evenly, and smartest optimizes for model capability scores based on historical performance data tracked in server/src/services/scoring.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}