# How Context Handoff Works in FreeLLMAPI: Preserving Conversation Continuity Across Model Switches

> Discover how FreeLLMAPI maintains conversation continuity with context handoff. Learn how synthesized system messages preserve session summaries across model switches without token overload.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: deep-dive
- Published: 2026-06-23

---

**FreeLLMAPI preserves conversation continuity when switching models by injecting a synthesized system message containing a truncated session summary, ensuring the new model receives critical context from previous turns without consuming excessive tokens.**

FreeLLMAPI is an open-source proxy that unifies multiple large language model providers under a single API. When users switch between models mid-conversation, the **context handoff** mechanism ensures the new model understands the conversation history without requiring clients to manually resend entire message threads.

## How Context Handoff Works in FreeLLMAPI

The context handoff system operates through six distinct phases implemented in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts). Each phase ensures that conversation state persists even when routing changes the underlying model provider.

### Tracking Session Messages

Every inbound request records the last approximately 12 messages (truncated to 500 characters each) in an in-memory `Map` keyed by the client-provided `sessionKey`. This logic resides in the `recordIncomingMessages` function (lines 56-79), which maintains a rolling buffer of conversation context accessible across different model calls.

### Recording the Last Successful Model

After a model completes a turn successfully, the service stores its identifier using the format `platform:modelId` in the same session entry. The `recordSuccessfulModel` function (lines 59-73) updates the `lastModelKey` property, creating a breadcrumb trail of which model handled the previous interaction.

### Detecting Model Switches

On subsequent requests, the proxy compares the stored `lastModelKey` against the model selected for the current turn. The `maybeInjectContextHandoff` function (lines 111-154) performs this comparison and determines whether a handoff is necessary based on the `FREELLMAPI_CONTEXT_HANDOFF` environment variable.

### Building a Concise Summary

When a switch is detected, the service invokes `buildSummary` (lines 94-100) to concatenate stored messages into a condensed narrative. This summary is capped at 6000 characters to balance context richness against token economy.

### Injecting the System-Message Handoff

The summary is wrapped in a system-role message beginning with `FreeLLMAPI context handoff:` and inserted after any existing system prompts in `maybeInjectContextHandoff` (lines 134-152). This positioning ensures provider-native system prompt ordering remains intact while giving the new model immediate awareness of the conversation trajectory.

### Token Head-Room Management

To prevent context window overflows, the proxy inflates routing token estimates by `HANDOFF_MAX_TOKENS` (approximately 1600 tokens) whenever a handoff might be injected. This padding, defined in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) (lines 21-24) and applied in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 54-60), reserves capacity for the injected system message.

## Configuration and Environment Variables

The entire flow is gated by the `FREELLMAPI_CONTEXT_HANDOFF` environment variable. When set to `on_model_switch`, handoff logic activates in `getContextHandoffMode` (lines 27-30). Any other value, or an unset variable, disables the feature entirely.

```bash

# .env or exported before starting the server

export FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

```

## Implementation Examples

### Initial Request with Session Tracking

Clients must provide a stable `sessionKey` to participate in handoff tracking. This identifier acts as the primary key for the in-memory message store.

```typescript
import axios from 'axios';

// The `sessionKey` is a stable identifier for a user conversation (e.g. a UUID).
const payload = {
  model: 'gpt-4o',          // first turn – model A
  messages: [{ role: 'user', content: 'Explain quantum tunnelling.' }],
  sessionKey: '123e4567-e89b-12d3-a456-426614174000',
};

await axios.post('https://api.freellmapi.com/v1/chat/completions', payload);

```

### Switching Models Mid-Conversation

When submitting a follow-up request to a different model, include the same `sessionKey`. FreeLLMAPI detects the switch and injects the context handoff automatically.

```typescript
const secondPayload = {
  model: 'anthropic/claude-3',  // model B – different platform
  messages: [
    { role: 'assistant', content: '... (previous answer)' },
    { role: 'user', content: 'Now apply that to a semiconductor device.' },
  ],
  sessionKey: '123e4567-e89b-12d3-a456-426614174000',
};

await axios.post('https://api.freellmapi.com/v1/chat/completions', secondPayload);

```

When processing the second request, FreeLLMAPI detects that `lastModelKey` (`gpt-4o`) differs from the selected model (`anthropic/claude-3`). The proxy injects a system message similar to:

```text
FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (gpt-4o → anthropic/claude-3).
Continue the user's task using the conversation context already provided in this request.
Do not restart the task, re‑ask already answered setup questions, or discard prior tool results.
Respect the user's latest message as the highest‑priority instruction.

Recent session summary:
User: Explain quantum tunnelling.
Assistant: ... (truncated)
User: Now apply that to a semiconductor device.

```

### Debugging Session Storage

For testing purposes, you can inspect or clear the internal store using the testing utility.

```typescript
import { _clearStoreForTesting } from './services/context-handoff.js';

// In a test environment you can clear the map:
_clearStoreForTesting();

```

## Key Implementation Files

The context handoff system spans several files in the FreeLLMAPI repository:

- **[`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts)**: Core logic for mode handling, message trimming, store management, injection, and token accounting.
- **[`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts)**: Where the handoff is optionally injected into outgoing requests and where token padding (`HANDOFF_MAX_TOKENS`) is applied.
- **[`server/src/services/request-retention.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/request-retention.ts)**: Provides the `sessionKey` plumbing used by `recordIncomingMessages`.
- **[`server/src/services/model-listing.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/model-listing.ts)**: Supplies the model-selection pipeline that yields the `selectedModelKey` used for handoff detection.

## Summary

- **Context handoff** activates when `FREELLMAPI_CONTEXT_HANDOFF` is set to `on_model_switch` and the `lastModelKey` differs from the current selection.
- The system stores approximately 12 recent messages per `sessionKey` in an in-memory Map, truncating each to 500 characters.
- `maybeInjectContextHandoff` in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) handles detection and injection logic (lines 111-154).
- A 6000-character summary caps the injected context, with `HANDOFF_MAX_TOKENS` (1600 tokens) reserved in the token budget.
- The handoff message is positioned after existing system prompts to respect provider-native ordering.

## Frequently Asked Questions

### What triggers a context handoff in FreeLLMAPI?

A context handoff triggers when two conditions align: the `FREELLMAPI_CONTEXT_HANDOFF` environment variable is set to `on_model_switch`, and the model selected for the current request differs from the `lastModelKey` stored in the session Map. The `maybeInjectContextHandoff` function performs this detection in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) (lines 111-154).

### How does FreeLLMAPI manage token limits during context handoff?

The proxy preemptively allocates `HANDOFF_MAX_TOKENS` (approximately 1600 tokens) to the routing budget whenever handoff is possible, as implemented in [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) (lines 54-60). Additionally, the `buildSummary` function limits the injected summary to 6000 characters, ensuring the downstream model's context window remains sufficient for the actual response generation.

### Can I disable context handoff if my application doesn't switch models?

Yes. Set the `FREELLMAPI_CONTEXT_HANDOFF` environment variable to any value other than `on_model_switch`, or leave it unset entirely. The `getContextHandoffMode` function (lines 27-30 in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts)) returns the active mode, and any non-matching value disables the injection logic entirely while maintaining normal proxy functionality.

### Does context handoff work across different LLM providers?

Yes. The mechanism is provider-agnostic. The `lastModelKey` stores identifiers in `platform:modelId` format (for example, `openai:gpt-4o` or `anthropic:claude-3`), and the handoff summary is injected as a standard system message compatible with OpenAI, Anthropic, and other supported API formats.