# How FreeLLMAPI Sticky Sessions Maintain Model Consistency in Conversations

> Discover how FreeLLMAPI sticky sessions leverage an in-memory cache to ensure consistent LLM model responses in multi-turn conversations, expiring after 30 minutes.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: deep-dive
- Published: 2026-06-25

---

**FreeLLMAPI binds each conversation to a specific LLM provider using an in-memory sticky session cache keyed by session identifiers, ensuring multi-turn exchanges remain consistent while automatically expiring entries after 30 minutes.**

FreeLLMAPI is an open-source request router that distributes traffic across multiple language model providers. To prevent conversation fragmentation—where sequential user messages hit different models—the codebase implements **sticky sessions** that remember which provider handled the previous turn. This mechanism, found in `tashfeenahmed/freellmapi`, guarantees model consistency while still permitting automatic fail-over when providers become unavailable.

## Session Identification and Key Composition

FreeLLMAPI derives a unique session identifier for every conversation thread using a two-tier fallback strategy documented at line 534 of [`README.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/README.md).

### Explicit Session Headers

Clients may supply a custom **`X-Session-Id`** header to maintain continuity across disjointed HTTP requests. When present, this value becomes the canonical **session key** for all subsequent sticky lookups.

### Deterministic Hash Fallback

If the header is absent, the router automatically computes the session key as the SHA-1 hash of the first user message in the conversation array. This deterministic approach ensures that identical initial prompts map to the same sticky slot without requiring client-side state management.

### Composite Sticky Keys

To prevent strategy bleed, the router constructs a **sticky strategy key** by concatenating the session key with the routing strategy name:

```typescript
stickyKey = sessionKey + ':' + strategyKey

```

Here, `strategyKey` represents the active routing mode (e.g., `model_group`, `fallback`). This isolation guarantees that a sticky entry created for one routing context does not influence another.

## Sticky Session Lookup and Storage Mechanism

The sticky session implementation resides primarily in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), which exposes two critical functions: `getStickyModel` and `setStickyModel`.

### Retrieving Sticky Entries

When processing a request, the router invokes `getStickyModel(messages, sessionIdHeader, stickyKey)` to query the global **`stickySessionMap`**. This `Map<string, {modelDbId: number; lastUsed: number}>` checks whether a valid entry exists for the computed `stickyKey`.

If an entry exists and its `lastUsed` timestamp is newer than **`STICKY_TTL_MS`** (30 minutes), the stored `modelDbId` is returned as the preferred provider for the current turn, bypassing normal priority selection.

### Recording Successful Responses

After a provider returns a valid response, the router calls `setStickyModel(messages, modelDbId, sessionIdHeader, stickyKey)` to persist the association. This operation writes the provider's database ID and current timestamp into `stickySessionMap`, ensuring the next turn targets the same model.

## Memory Management and TTL Constraints

The sticky cache implementation in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) enforces strict resource limits to prevent unbounded memory growth.

### 30-Minute Expiration Window

The constant **`STICKY_TTL_MS`** defines a hard 30-minute expiration. Before every routing decision, the code iterates the `stickySessionMap` and deletes entries where `Date.now() - lastUsed` exceeds this threshold. This guarantees that abandoned conversations do not consume slots indefinitely.

### LRU Eviction at 500 Entries

The map is capped at **500 entries**. When insertion would exceed this limit, the router evicts the oldest entry based on the `lastUsed` timestamp, implementing a simple least-recently-used (LRU) strategy.

```typescript
// Cleanup excerpt from server/src/routes/proxy.ts
const now = Date.now();
for (const [k, v] of stickySessionMap) {
  if (now - v.lastUsed > STICKY_TTL_MS) stickySessionMap.delete(k);
}
if (stickySessionMap.size > 500) {
  let oldestKey: string | undefined;
  let oldest = Infinity;
  for (const [k, v] of stickySessionMap) {
    if (v.lastUsed < oldest) {
      oldest = v.lastUsed;
      oldestKey = k;
    }
  }
  if (oldestKey) stickySessionMap.delete(oldestKey);
}

```

## Integration with Routing Strategies

Sticky sessions interact with other routing features to respect provider capabilities while maintaining consistency.

### Vision Filtering

When a request contains image inputs, the router validates whether the sticky model supports vision capabilities. If the cached provider lacks image support, the sticky entry is ignored for that turn, allowing the router to select a vision-compatible alternative without corrupting the session cache.

### Group-Pinned Sessions

Users may pin a specific model group via configuration. When active, the sticky entry is written using the same `stickyKey` composition, ensuring subsequent turns read from the exact group membership, effectively overriding the default priority order while maintaining stickiness within the pinned set.

### Fail-Over Behavior

If the sticky model provider is temporarily unavailable or returns an error, the router treats the lookup as a cache miss and proceeds with standard fail-over logic. Importantly, the stale entry is **not** immediately purged; it remains in `stickySessionMap` until the 30-minute TTL expires, allowing the provider to recover without losing session affinity.

## Extended Context Hand-Off

For conversations that pause longer than the standard TTL, [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) maintains a parallel **hand-off map** with a 3-hour retention period. This secondary cache mirrors sticky entries, enabling the system to recall the last-used model for a session even after the primary 30-minute window lapses, provided the conversation resumes within the extended window.

## Summary

- FreeLLMAPI generates session keys from the **`X-Session-Id`** header or a SHA-1 hash of the first message.
- Composite sticky keys isolate routing strategies to prevent cross-contamination.
- The **`stickySessionMap`** in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) stores model bindings with a 30-minute TTL and 500-entry LRU cap.
- Functions **`getStickyModel`** and **`setStickyModel`** in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) handle cache retrieval and persistence.
- Vision requests bypass incompatible sticky models without destroying the session affinity.
- A secondary 3-hour hand-off cache in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) preserves context across longer interruptions.

## Frequently Asked Questions

### How does FreeLLMAPI identify a conversation session?

FreeLLMAPI checks for an `X-Session-Id` header first. If missing, it falls back to computing a SHA-1 hash of the first user message content to generate a deterministic identifier, as noted at line 534 of the [`README.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/README.md).

### What happens when a sticky model doesn't support vision inputs?

The router detects vision-capable requests and validates the sticky model against the required features. If the cached model lacks image support, the sticky entry is skipped for that specific turn, allowing selection of a compatible provider while preserving the original entry for future text-only turns.

### How long does FreeLLMAPI remember a sticky session?

Primary sticky entries persist for **30 minutes** (`STICKY_TTL_MS`) from the last usage timestamp. Additionally, [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) maintains a secondary mirror with a **3-hour** retention period for extended conversation recovery.

### What occurs if the sticky model provider becomes unavailable?

The router treats unavailable sticky models as cache misses and executes normal fail-over routing to the next eligible provider. The original sticky entry remains in the map until its TTL expires, allowing automatic reversion if the provider recovers within the 30-minute window.