# How FreeLLMAPI Manages Context Handoff When Switching Models Mid-Conversation

> Discover how FreeLLMAPI seamlessly switches models mid-conversation. Learn about its context handoff mechanism and session store for uninterrupted AI interactions.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-06-29

---

**FreeLLMAPI maintains a lightweight per-session store that records recent conversation turns and the previous model identifier, automatically injecting a summary system message when `FREELLMAPI_CONTEXT_HANDOFF=on_model_switch` detects a provider change.**

FreeLLMAPI is an open-source LLM gateway that enables dynamic routing between multiple AI providers without losing conversational continuity. When users switch models mid-conversation—transitioning from GPT-4 to Claude or any other supported backend—the system must ensure the new model comprehends the prior context without requiring client-side state management. The repository solves this through a server-side context handoff mechanism implemented in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts).

## The Context Handoff Architecture

The implementation relies on a **stateless API surface** with bounded server-side storage. Rather than forcing clients to resubmit entire conversation histories or maintain complex state, FreeLLMAPI tracks three critical pieces of data per session: the last 12 conversation turns, the timestamp of the last activity, and the identifier of the model that processed the previous request. This design ensures that context handoff when switching models mid-conversation happens transparently, with the injected system message providing the new model with a concise summary of the discussion so far.

## Step-by-Step Implementation

The context handoff logic proceeds through three distinct phases, each handled by specific functions in the handoff service.

### 1. Recording Incoming Messages

When a request arrives at the proxy handler ([`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts)), the system immediately calls `recordIncomingMessages(sessionKey, messages)` to update the session store. This function performs aggressive trimming to control memory usage:

- Retains only the **last 12 turns** of the conversation
- Limits each message to **500 characters**
- Clears the `lastModelKey` field if the payload contains no assistant turns (indicating a fresh conversation)

This ensures the store never grows unbounded while preserving the most relevant recent context.

### 2. Tracking the Active Model

After the router ([`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)) successfully receives a response from the selected provider, it invokes `recordSuccessfulModel({sessionKey, modelKey})`. This writes the provider identifier (e.g., `"gpt-4"` or `"claude-2"`) into the session entry, enabling the next request to detect whether a different model has been selected.

### 3. Injecting the Handoff Message

Before routing a request, the system calls `maybeInjectContextHandoff({mode, sessionKey, messages, selectedModelKey})`. If the environment variable `FREELLMAPI_CONTEXT_HANDOFF` is set to `"on_model_switch"` and the stored `lastModelKey` differs from the newly selected model, the function:

1. Generates a summary via `buildSummary()`, capped at **6,000 characters**
2. Constructs a system message with the template: `FreeLLMAPI context handoff: You are taking over an ongoing conversation from another model (<prev> → <new>)...`
3. Inserts the message immediately after any existing system messages, preserving provider-specific prompt ordering
4. Returns the estimated token count (`injectedTokens`) alongside the modified message array

The function also checks for duplicate handoff messages (`alreadyPresent`) to prevent redundant injection if the user manually included a similar instruction.

## Configuration and Environment Variables

The entire handoff mechanism is gated by the `getContextHandoffMode()` function, which reads the environment at runtime:

```typescript
export function getContextHandoffMode(): ContextHandoffMode {
  const raw = process.env.FREELLMAPI_CONTEXT_HANDOFF?.trim().toLowerCase();
  return raw === 'on_model_switch' ? 'on_model_switch' : 'off';
}

```

When the mode is `"off"` (the default), the system skips injection entirely and passes requests through unchanged. To enable automatic context handoff when switching models mid-conversation, set:

```bash
FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

```

## Integration Points in the Request Lifecycle

The handoff service integrates at three critical points in the request pipeline:

- **Entry point**: The proxy handler ([`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts)) calls `recordIncomingMessages` immediately upon receiving the user payload, ensuring the store stays current regardless of routing decisions
- **Pre-routing**: The router ([`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts)) invokes `maybeInjectContextHandoff` before forwarding requests to the selected provider, allowing the injected summary to reach the new model
- **Post-response**: The router calls `recordSuccessfulModel` after receiving a successful provider response, updating the session with the model that just completed the turn

## Safety Mechanisms and Token Budgeting

FreeLLMAPI implements several safeguards to prevent the handoff mechanism from overwhelming models or exhausting server resources:

- **Token budgeting**: The exported constant `HANDOFF_MAX_TOKENS` (approximately 1,600 tokens) is added to the router's token estimate, ensuring the injected summary never pushes the request beyond the model's context window
- **Memory bounds**: The session store caps at **500 active sessions** and evicts entries after a **3-hour TTL**, preventing unbounded growth
- **Deduplication**: The injection logic detects existing handoff messages to avoid duplicate context summaries

## Code Examples

### Enabling Context Handoff

Set the environment variable before starting the server:

```bash

# .env

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

```

### Recording User Turns

Call this when handling incoming chat requests:

```typescript
import { recordIncomingMessages } from '@freellmapi/server/src/services/context-handoff';

// sessionKey derived from cookie or Authorization header
recordIncomingMessages(sessionKey, incomingChatMessages);

```

### Checking and Injecting Context

Use this in your request router before forwarding to the provider:

```typescript
import { maybeInjectContextHandoff, getContextHandoffMode } from '@freellmapi/server/src/services/context-handoff';

const mode = getContextHandoffMode();
const { messages, injected, injectedTokens } = maybeInjectContextHandoff({
  mode,
  sessionKey,
  messages: originalMessages,
  selectedModelKey: providerModelKey, // e.g., "gpt-4", "claude-2"
});

```

### Updating the Model Tracker

Call after receiving a successful response:

```typescript
import { recordSuccessfulModel } from '@freellmapi/server/src/services/context-handoff';

recordSuccessfulModel({ sessionKey, modelKey: providerModelKey });

```

### Reserving Token Budget

Account for handoff overhead in your token calculations:

```typescript
import { HANDOFF_MAX_TOKENS } from '@freellmapi/server/src/services/context-handoff';

const tokenBudget = modelContextWindow - requestTokens - HANDOFF_MAX_TOKENS;

```

## Summary

- **FreeLLMAPI** manages context handoff when switching models mid-conversation through a server-side session store that tracks the last 12 turns and the previous model identifier
- The **`recordIncomingMessages`** function in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) maintains the conversation buffer with strict limits (12 turns, 500 chars per message)
- **`maybeInjectContextHandoff`** automatically inserts a summary system message when it detects a model change and the `FREELLMAPI_CONTEXT_HANDOFF` environment variable is set to `"on_model_switch"`
- The **`HANDOFF_MAX_TOKENS`** constant (~1,600 tokens) ensures the injected summary never exceeds the model's context window
- The store implements **TTL-based eviction** (3 hours) and caps at 500 sessions to prevent memory leaks

## Frequently Asked Questions

### What happens if context handoff is disabled?

When `FREELLMAPI_CONTEXT_HANDOFF` is unset or set to `"off"`, the `getContextHandoffMode()` function returns `"off"` and `maybeInjectContextHandoff` returns the original messages unchanged. The system passes requests directly to the new model without injecting summary context, meaning the model receives only the conversation history explicitly sent by the client.

### How does FreeLLMAPI prevent the handoff message from exceeding token limits?

The system uses the exported constant `HANDOFF_MAX_TOKENS` (approximately 1,600 tokens) as a safety margin. The router subtracts this value from the available context window before checking if the request fits within the model's limits. Additionally, the `buildSummary` function caps the generated summary at 6,000 characters, and individual stored messages are truncated to 500 characters each.

### Can the context handoff message template be customized?

Currently, the handoff message template is hardcoded within the `maybeInjectContextHandoff` function in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts). The template includes the previous and new model names, specific instructions for the incoming model, and the recent session summary. Users cannot modify this template via configuration files; customization would require editing the source code and rebuilding the service.

### Where is the conversation history stored during a session?

The conversation history resides in a server-side in-memory store maintained by the context handoff service. This store caps at 500 concurrent sessions and automatically evicts entries after 3 hours of inactivity. The client never needs to maintain this state, enabling truly stateless API interactions while preserving continuity across model switches.