# How Context Handoff Works When Switching Models Mid-Conversation in FreeLLMAPI

> Discover how FreeLLMAPI ensures seamless context handoff when switching models mid-conversation. Maintain continuity and an uninterrupted user experience with our innovative solution.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-28

---

**FreeLLMAPI preserves conversation continuity by injecting a summary system message when the environment variable `FREELLMAPI_CONTEXT_HANDOFF` is set to `on_model_switch`, comparing the last used model with the newly selected one and automatically briefing the new model on recent turns.**

Maintaining context across different language models is challenging when routing requests dynamically. The **FreeLLMAPI** repository solves this through a server-side **context handoff** mechanism that activates when switching models mid-conversation, eliminating the need for client-side changes while preventing context window overflow.

## The Three-Step Handoff Pipeline

The implementation in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) manages context preservation through three coordinated operations that execute during the request lifecycle.

### Step 1: Recording Incoming Messages

When a chat request arrives, the proxy handler immediately calls `recordIncomingMessages(sessionKey, messages)` to update the session store. This function caps the history at the last **12 turns**, truncates each message to **500 characters**, and stores them with a timestamp. If the payload contains no assistant turns—indicating a fresh conversation—the function clears the `lastModelKey` field to reset the handoff state.

### Step 2: Tracking the Last Successful Model

After a provider successfully returns a response, the router invokes `recordSuccessfulModel({ sessionKey, modelKey })` to write the model identifier (e.g., `"gpt-4"` or `"claude-2"`) into the session entry. This stored key serves as the comparison point for the next request to determine if a model switch occurred.

### Step 3: Conditional Injection of Handoff Messages

Before routing a request to the selected provider, the system calls `maybeInjectContextHandoff({ mode, sessionKey, messages, selectedModelKey })`. When `mode` is `'on_model_switch'` and the stored `lastModelKey` differs from the `selectedModelKey`, the function generates a summary via `buildSummary` of the stored recent messages (capped at **6,000 characters**) and inserts a system message immediately after any leading system messages. The injected message follows this format:

```

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (<prev> → <new>).
... (instructions) ...
Recent session summary:
User: ...
Assistant: ...

```

The function returns `injectedTokens`, which the router adds to its token budget calculations using the exported constant `HANDOFF_MAX_TOKENS` (approximately **1,600 tokens**).

## Configuration and Activation

Context handoff is disabled by default. To enable the feature, set the environment variable before starting the server:

```bash
FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

```

The `getContextHandoffMode()` function in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) reads this variable, trims and lowercases the value, and returns either `'on_model_switch'` or `'off'`. When set to `'off'`, the middleware skips all injection logic and passes requests through unchanged.

## Integration Points in the Request Lifecycle

The handoff mechanism integrates at three critical points in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) and [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts):

1. **Request ingestion**: The proxy handler calls `recordIncomingMessages` as soon as the payload arrives, ensuring the session store reflects the latest user input regardless of which model will handle it.

2. **Pre-routing**: The router executes `maybeInjectContextHandoff` to check for model switches and potentially inject the summary message before forwarding to the provider.

3. **Post-response**: After receiving a successful provider response, the router calls `recordSuccessfulModel` to update the session with the model that just completed the turn.

## Safety Mechanisms and Memory Limits

FreeLLMAPI implements several safeguards to prevent unbounded growth and token overflow:

- **Session cap**: The store limits active sessions to **500 entries**, evicting the oldest after a **3-hour TTL**.
- **Deduplication**: The injection logic detects if the provider already received a hand-off system message (e.g., manually added by the user) via the `alreadyPresent` check, preventing duplicates.
- **Token budgeting**: The constant `HANDOFF_MAX_TOKENS` reserves headroom in the routing token estimate, ensuring the injected summary never exceeds the model's context window.

## Implementation Examples

### Enabling Context Handoff

Set the environment variable in your `.env` file or runtime environment:

```bash
FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

```

### Recording User Turns

Import and call the recording function in your request handler:

```typescript
import { recordIncomingMessages } from '@freellmapi/server/src/services/context-handoff';

// sessionKey derives from a cookie or Authorization header
recordIncomingMessages(sessionKey, incomingChatMessages);

```

### Injecting Handoff Messages Before Routing

Check the mode and conditionally inject context when switching models:

```typescript
import { maybeInjectContextHandoff, getContextHandoffMode } from '@freellmapi/server/src/services/context-handoff';

const mode = getContextHandoffMode();
const { messages, injected, injectedTokens } = maybeInjectContextHandoff({
  mode,
  sessionKey,
  messages: originalMessages,          // payload from the client
  selectedModelKey: providerModelKey,  // e.g., "gpt-4", "claude-2"
});

```

### Tracking Model Success

After a successful response, update the session to remember which model handled the turn:

```typescript
import { recordSuccessfulModel } from '@freellmapi/server/src/services/context-handoff';

recordSuccessfulModel({ sessionKey, modelKey: providerModelKey });

```

### Reserving Token Budget

Use the exported constant to account for potential handoff overhead:

```typescript
import { HANDOFF_MAX_TOKENS } from '@freellmapi/server/src/services/context-handoff';

const tokenBudget = modelContextWindow - requestTokens - HANDOFF_MAX_TOKENS;

```

## Summary

- **FreeLLMAPI** enables seamless **context handoff when switching models mid-conversation** through the `FREELLMAPI_CONTEXT_HANDOFF` environment variable set to `on_model_switch`.
- The system stores the last 12 turns (500 characters each) in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts), tracking which model last responded via `recordSuccessfulModel`.
- When models differ, `maybeInjectContextHandoff` automatically inserts a summary system message capped at 6,000 characters and approximately 1,600 tokens.
- Integration occurs in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (recording input) and [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) (injection and tracking).
- Safety mechanisms include a 500-session limit with 3-hour TTL, deduplication checks, and token budget reservation via `HANDOFF_MAX_TOKENS`.

## Frequently Asked Questions

### How does FreeLLMAPI detect when to trigger a context handoff?

FreeLLMAPI compares the `lastModelKey` stored in the session with the `selectedModelKey` of the incoming request. If the environment variable `FREELLMAPI_CONTEXT_HANDOFF` is set to `on_model_switch` and these keys differ, the `maybeInjectContextHandoff` function automatically generates a summary message. This detection happens server-side in the router before forwarding the request to the new provider.

### What happens if context handoff is disabled?

When `FREELLMAPI_CONTEXT_HANDOFF` is set to `off` or undefined, `getContextHandoffMode()` returns `'off'`, causing `maybeInjectContextHandoff` to skip all injection logic. The request passes through unchanged, meaning the new model receives no summary of previous turns, effectively starting a fresh conversation from that model's perspective.

### How does the system prevent duplicate handoff messages?

The `maybeInjectContextHandoff` function checks for an `alreadyPresent` flag that detects if the messages array already contains a handoff system message (such as one manually added by the client). If detected, the function avoids injecting a duplicate summary. Additionally, the system inserts the handoff message immediately after any existing system messages to preserve provider-specific prompt ordering.

### What are the memory and token limits for stored context?

The session store caps history at **12 turns** with **500 characters per message**, and the generated summary is limited to **6,000 characters**. The system maintains a maximum of **500 active sessions** with a **3-hour TTL** to prevent memory leaks. Token budgeting reserves approximately **1,600 tokens** via `HANDOFF_MAX_TOKENS` to ensure the injected summary never exceeds the model's context window.