How Sticky Sessions Work in FreeLLMAPI: Technical Implementation Guide

FreeLLMAPI implements sticky sessions by persisting the model that successfully served the last assistant turn of a conversation and preferring that same model on subsequent requests for up to 30 minutes.

FreeLLMAPI is an open-source LLM routing proxy designed to maintain conversation continuity across multiple providers. The sticky session mechanism ensures that multi-turn conversations remain bound to the same model, preventing context fragmentation and inconsistent responses caused by mid-conversation provider switches.

Core Architecture of Sticky Sessions

The sticky session system consists of three tightly-coupled components that work together to maintain session stability.

Session Key Generation

The foundation of the sticky system is deterministic session identification. In server/src/routes/proxy.ts, the getSessionKey() function constructs a unique identifier using either the client's x-session-id header or a SHA-1 hash derived from the first user message combined with the routing strategy key.

When present, the x-session-id header takes precedence, producing keys like hdr:abc123::smart (where "smart" represents the routing strategy). Without a header, the system generates a hash-based key from the message content and strategy, ensuring identical conversations receive consistent routing even across reconnections.

The Sticky Map Storage

FreeLLMAPI maintains an in-memory sticky map defined in server/src/routes/proxy.ts as Map<string,{modelDbId,lastUsed}>. This map stores the model database ID alongside a timestamp of the last successful response.

The map enforces automatic expiration through STICKY_TTL_MS (set to 30 minutes) and implements memory safety constraints with a soft cap of 500 entries and a hard cap of 1000. When the hard limit triggers, the system evicts the oldest entries based on the lastUsed timestamp.

Router Integration

The routing layer in server/src/services/router.ts receives the sticky preference through the preferredModelDbId parameter in the routeRequest() function (lines 37-45). When a sticky model ID is present, the router either:

• Moves the existing model to the front of the sorted provider chain, or
• Injects a custom row for the model if it exists in the database but isn't part of the current active chain

This injection mechanism ensures that even custom or out-of-profile models can be honored for sticky session continuity.

Step-by-Step Execution Flow

The sticky session mechanism operates through a six-stage pipeline for each conversation turn.

1. Request Intake and Key Resolution

When a request arrives at the proxy, the system extracts the optional x-session-id header (sessionIdHeader) and invokes getSessionKey(messages, sessionIdHeader, strategyKey) to generate the session identifier.

2. Sticky Lookup

The getStickyModel() function checks if the conversation contains an existing assistant turn (hasAssistant). If the session key exists in stickySessionMap and Date.now() - entry.lastUsed <= STICKY_TTL_MS, the function returns the stored modelDbId.

3. Routing Preference

The router receives preferredModelDbId from the sticky lookup. If present, the router locates the model in the sorted chain and splices it to index 0, ensuring highest priority selection.

4. Model Execution

The request dispatches to the selected model provider.

5. Persistence

After a successful response, the proxy calls setStickyModel(messages, route.modelDbId, sessionIdHeader, stickyStrategyKey) in server/src/routes/proxy.ts. This updates the map with the current timestamp, resetting the 30-minute TTL window.

6. Expiration and Cleanup

On subsequent reads, expired entries (older than 30 minutes) are automatically removed from the map, causing the request to fall back to standard auto-routing logic.

Code Implementation Examples

Client-Side Session Initialization

const response = await fetch('https://api.freellmapi.com/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'x-session-id': 'my-conversation-123',   // Sticky session identifier
  },
  body: JSON.stringify({
    model: 'auto',
    messages: [{role:'user', content:'Hello!'}],
  }),
});

Proxy-Level Sticky Handling

// Inside server/src/routes/proxy.ts
const sticky = getStickyModel(messages, sessionIdHeader, stickyStrategyKey);
const preferredModelDbId = sticky ?? undefined;

// After successful model call:
setStickyModel(messages, route.modelDbId, sessionIdHeader, stickyStrategyKey);

Router Integration Logic

// Inside server/src/services/router.ts
if (preferredModelDbId) {
  const idx = sortedChain.findIndex(e => e.model_db_id === preferredModelDbId);
  if (idx >= 0) {
    const [preferred] = sortedChain.splice(idx, 1);
    sortedChain.unshift(preferred);
  } else {
    // Inject custom/pinned model not in current chain
    const pinnedRow = db.prepare(`
      SELECT m.id as model_db_id, m.platform, m.model_id, 
             m.display_name, m.intelligence_rank
      FROM models m
      WHERE m.id = ? AND m.enabled = 1
    `).get(preferredModelDbId);
    if (pinnedRow) sortedChain.unshift(pinnedRow);
  }
}

TTL and Memory Management

The sticky session system balances continuity with resource constraints through aggressive expiration policies. The 30-minute TTL (STICKY_TTL_MS) prevents stale sessions from consuming memory while allowing sufficient time for typical conversational interactions.

Memory protection operates on two levels: a soft cap at 500 entries triggers background pruning of expired items, while a hard cap at 1000 entries forces eviction of the oldest lastUsed entries regardless of TTL status. This ensures the proxy remains stable under high traffic volumes without manual intervention.

Summary

• Sticky sessions in FreeLLMAPI bind conversations to specific models using a deterministic session key and persistent map storage.
• The system lives primarily in server/src/routes/proxy.ts with routing integration in server/src/services/router.ts.
• Session keys derive from the x-session-id header or message content hashes combined with routing strategy.
• Entries persist for 30 minutes (STICKY_TTL_MS) with automatic cleanup via memory caps (500 soft, 1000 hard).
• The router honors sticky preferences by reordering or injecting the preferred model into the provider chain.

Frequently Asked Questions

What happens when a sticky session expires?

When the 30-minute TTL expires, getStickyModel() returns undefined, causing the router to ignore the stale preference and select a new model based on current availability, load, and routing strategy. The expired entry is automatically pruned from the map on the next access.

Can clients override sticky session behavior?

Yes. Clients can force a fresh model selection by omitting the x-session-id header and ensuring the first user message content differs from previous turns, which generates a new session key. Alternatively, sending a new unique x-session-id header effectively starts a new sticky session unrelated to previous conversations.

How does FreeLLMAPI handle sticky sessions when the preferred model becomes unavailable?

If the sticky model ID specified in preferredModelDbId is not found in the active chain, the router attempts to inject it by querying the database for enabled = 1 models. If the model is disabled or deleted, the injection fails silently, and the router proceeds with the standard sorted chain, effectively falling back to auto-routing for that turn.

What is the performance impact of sticky session lookups?

The sticky map operates as an in-memory JavaScript Map with O(1) lookup complexity. According to the source code in server/src/routes/proxy.ts, the lookup involves a single get() operation and a timestamp comparison, adding negligible latency (microseconds) to the request processing pipeline compared to the network overhead of LLM API calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}