How Sticky Sessions Keep Conversations on the Same Model in FreeLLMAPI

FreeLLMAPI's sticky sessions feature binds each conversation to a specific LLM model using a deterministic session key and an in-memory cache with a 30-minute TTL, ensuring consistent responses and preventing mid-conversation model switching.

The open-source FreeLLMAPI project (tashfeenahmed/freellmapi) implements a lightweight sticky session mechanism that maintains conversation continuity across multiple API calls. By caching the model assignment in an in-process Map, the router can return subsequent messages to the same model that handled the initial request, avoiding the hallucinations and inconsistency that often occur when models switch mid-conversation.

Session Key Generation

Every sticky session begins with a deterministic session key that identifies the conversation. In server/src/services/router.ts, the getSessionKey function (lines 31-42) constructs this key using one of two strategies:

• Explicit header: If the client sends an X-Session-Id header, the router uses that value (optionally combined with a routing strategy key) as the session identifier.
• Content hash: When no header is present, the router generates a SHA-1 hash of the first user message text to create the key.

// server/src/routes/proxy.ts
const key = getSessionKey(messages, sessionIdHeader, strategyKey);

This deterministic approach ensures that rerunning the same conversation without a session header still maps to the same sticky entry, while explicit headers allow clients to maintain continuity across disjointed message sequences.

The Sticky Session Map

The core storage mechanism is a module-level Map defined in server/src/routes/proxy.ts (lines 25-30):

// server/src/routes/proxy.ts
const stickySessionMap = new Map<string, { modelDbId: number; lastUsed: number }>();
const STICKY_TTL_MS = 30 * 60 * 1000; // 30 minutes

Each entry stores:

• modelDbId: The database identifier of the LLM model that last served this session
• lastUsed: A timestamp tracking the last successful request

The map enforces a hard limit of 500 entries and a 30-minute TTL to prevent unbounded memory growth while keeping recent sessions active.

Reading and Writing Sticky Entries

Before routing a request, the system checks for an existing sticky assignment. The getStickyModel function in server/src/services/router.ts (lines 44-58) validates the TTL and returns the stored modelDbId only if the entry is still fresh:

// server/src/routes/proxy.ts
const sticky = getStickyModel(messages, sessionIdHeader, stickyStrategyKey);
const preferredModel =
    (sticky != null && groupChain.some(r => r.model_db_id === sticky))
      ? sticky
      : undefined;

After a successful routing decision, setStickyModel (lines 61-72) records the model that served the request:

// server/src/routes/proxy.ts
setStickyModel(messages, route.modelDbId, sessionIdHeader, stickyStrategyKey);

This write operation updates the timestamp and performs housekeeping—evicting entries that exceed the 500-entry limit or have passed the TTL threshold.

Integration with Routing Logic

The sticky model ID flows through the routing pipeline as preferredModelDbId. According to the source code comments in server/src/services/router.ts (lines 95-99):

"If preferredModelDbId is set, that model gets tried FIRST (sticky sessions). This prevents hallucination from model switching mid-conversation."

The routeRequest function receives this preference and prioritizes the matching model in the candidate chain. If the preferred model is unavailable or the sticky entry has expired, the router falls back to standard load-balancing logic.

Practical Usage Examples

Using an Explicit Session Header

Provide an X-Session-Id header to maintain continuity across API calls:

import axios from 'axios';

await axios.post(
  'https://api.example.com/v1/chat/completions',
  {
    messages: [{ role: 'user', content: 'Hello, who are you?' }],
  },
  {
    headers: {
      'X-Session-Id': 'conversation-12345',
    },
  },
);

The first request creates a sticky entry keyed to conversation-12345. Subsequent calls reusing this header route to the same model until 30 minutes of inactivity passes.

Hash-Based Sessions (No Header)

Allow the system to generate a session key from message content:

await axios.post(
  'https://api.example.com/v1/chat/completions',
  {
    messages: [
      { role: 'user', content: 'Explain quantum tunneling.' },
    ],
  },
);

Without the X-Session-Id header, the router hashes the first user message ("Explain quantum tunneling."). All turns containing this identical first message share the same sticky model assignment.

Debugging Sticky Sessions

Inspect the internal map for development purposes:

import { stickySessionMap } from './server/src/routes/proxy.ts';

for (const [key, { modelDbId, lastUsed }] of stickySessionMap) {
  console.log(`Key ${key} → Model ${modelDbId}, last used ${new Date(lastUsed)}`);
}

Note: This map is not exported in production builds and should only be used in development environments.

Summary

• Sticky sessions in FreeLLMAPI use a deterministic key (either X-Session-Id header or SHA-1 hash of the first message) to identify conversations.
• The stickySessionMap in server/src/routes/proxy.ts caches the model assignment for up to 30 minutes with a maximum of 500 entries.
• The getStickyModel and setStickyModel functions in server/src/services/router.ts handle TTL validation and cleanup, ensuring memory efficiency.
• The preferred model is tried first in the routing chain, falling back to normal selection only if the sticky entry expires or the model is unavailable.

Frequently Asked Questions

How does FreeLLMAPI generate session keys without explicit headers?

When the X-Session-Id header is absent, the getSessionKey function computes a SHA-1 hash of the first user message's content. This hash becomes the session key, allowing conversations to remain sticky even when clients cannot track session IDs themselves, provided the initial message content remains consistent.

What happens when the sticky session map reaches 500 entries?

When the map exceeds 500 entries, setStickyModel performs housekeeping by scanning and removing entries older than the 30-minute TTL. This bounded size prevents memory leaks in high-throughput environments while maintaining recent session affinity.

How long do sticky sessions persist?

Sticky entries expire after 30 minutes of inactivity (controlled by STICKY_TTL_MS in server/src/routes/proxy.ts). After expiration, the next request for that session generates a new sticky assignment, potentially routing to a different model.

Can I disable sticky sessions for specific requests?

There is no explicit disable flag in the current implementation. To effectively bypass sticky sessions, omit the X-Session-Id header and vary the first user message content, which forces a new session key generation and prevents cache hits in the stickySessionMap.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}