# How FreeLLMAPI Implements Context Handoff When Switching Models Mid-Session

> Learn how FreeLLMAPI enables seamless context handoff mid-conversation. Discover its session cache and system message technique for continuous model switching.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: internals
- Published: 2026-06-25

---

**FreeLLMAPI maintains continuity across model switches by storing recent conversation turns in a lightweight session cache and automatically injecting a contextual summary system message when a different model takes over the next turn.**

FreeLLMAPI is an open-source proxy that routes LLM requests to multiple providers, and it solves the critical problem of **context handoff** when users switch between models mid-conversation. Rather than forcing clients to manage state or resend entire histories, the server intercepts model switches and synthesizes a handoff message that compresses recent context for the new model.

## The Three-Step Handoff Mechanism

The context handoff implementation lives primarily in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) and operates through three coordinated stages that track conversation state and trigger summary injection.

### Recording Incoming Messages

When a request first enters the system, the proxy handler immediately calls `recordIncomingMessages(sessionKey, messages)` to update the session store. This function trims the payload to the last **12 turns** and limits each message to **500 characters**, storing them with a timestamp. If the payload contains no assistant turns—indicating a fresh conversation—it clears the `lastModelKey` field to reset the handoff state.

```typescript
import { recordIncomingMessages } from '@freellmapi/server/src/services/context-handoff';

// Called at the start of every request in server/src/routes/proxy.ts
recordIncomingMessages(sessionSessionKey, incomingChatMessages);

```

The session store enforces hard limits: it caps at **500 sessions** and evicts entries older than **3 hours**, ensuring the cache never grows unbounded.

### Tracking the Last Successful Model

After a provider successfully returns a response, the router calls `recordSuccessfulModel({ sessionKey, modelKey })` to record which model handled the turn. This writes the model key (e.g., `"gpt-4"` or `"claude-2"`) into the session entry, creating a breadcrumb that the next request can compare against.

```typescript
import { recordSuccessfulModel } from '@freellmapi/server/src/services/context-handoff';

// Called after successful provider response in server/src/services/router.ts
recordSuccessfulModel({ sessionKey, modelKey: providerModelKey });

```

### Injecting the Handoff Summary

Before routing a request to the selected provider, the system invokes `maybeInjectContextHandoff({ mode, sessionKey, messages, selectedModelKey })`. When the environment variable `FREELLMAPI_CONTEXT_HANDOFF` is set to `"on_model_switch"` and the stored `lastModelKey` differs from the newly selected model, the function builds a conversation summary using `buildSummary()`—capped at **6,000 characters**—and inserts a system message immediately after any existing system prompts.

The injected message follows this structure:

```

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (<previous_model> → <new_model>).
... (instructions) ...
Recent session summary:
User: ...
Assistant: ...

```

The function returns `injectedTokens` (estimated using the exported constant `HANDOFF_MAX_TOKENS`, roughly **1,600 tokens**), which the router adds to its token budget calculations to prevent context window overflows.

```typescript
import { maybeInjectContextHandoff, getContextHandoffMode } from '@freellmapi/server/src/services/context-handoff';

const mode = getContextHandoffMode(); // Reads FREELLMAPI_CONTEXT_HANDOFF env var
const { messages, injected, injectedTokens } = maybeInjectContextHandoff({
  mode,
  sessionKey,
  messages: originalMessages,
  selectedModelKey: providerModelKey,
});

```

## Configuration and Environment Setup

The entire handoff mechanism is gated by the `FREELLMAPI_CONTEXT_HANDOFF` environment variable. The function `getContextHandoffMode()` in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) reads this variable and returns either `'on_model_switch'` or `'off'`. When set to `'off'`, the system skips injection entirely and passes requests through unchanged.

```bash

# Enable automatic context handoff on model switches

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

```

This design keeps the feature opt-in and ensures zero overhead when disabled.

## Integration with the Request Lifecycle

The handoff logic integrates at two critical points in the request pipeline to maintain stateless operation while preserving context across turns.

### Proxy Handler Initialization

In [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts), the incoming request handler immediately records the user's messages via `recordIncomingMessages`. This ensures the session cache stays current regardless of which model ultimately processes the request or whether the request succeeds.

### Router Orchestration

The [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) file orchestrates the handoff check. Before forwarding to the selected provider, it calls `maybeInjectContextHandoff` to potentially prepend the summary. After receiving a successful response, it updates the model tracking via `recordSuccessfulModel`. This creates a closed loop where every turn updates the session state for the next.

## Safety Mechanisms and Limits

FreeLLMAPI implements several safeguards to ensure context handoff never degrades performance or breaks provider limits:

- **Token Budget Safety**: The exported constant `HANDOFF_MAX_TOKENS` (~1,600 tokens) is subtracted from the model's available context window during routing calculations, guaranteeing the injected summary never pushes requests over the limit.
- **Duplicate Prevention**: The injection logic checks for existing handoff messages (`alreadyPresent`) to avoid duplicating summaries if a user manually includes one or if the same request is retried.
- **Bounded Storage**: The session store's 500-entry cap and 3-hour TTL prevent memory leaks in long-running server instances.
- **Graceful Degradation**: If the session store is unavailable or the summary generation fails, the system falls back to sending the original messages without injection, preserving availability over context continuity.

```typescript
import { HANDOFF_MAX_TOKENS } from '@freellmapi/server/src/services/context-handoff';

// Reserve headroom for potential handoff injection
const tokenBudget = modelContextWindow - requestTokens - HANDOFF_MAX_TOKENS;

```

## Summary

- **FreeLLMAPI** implements context handoff through a lightweight session cache in [`server/src/services/context-handoff.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/context-handoff.ts) that stores the last 12 conversation turns.
- **Automatic injection** triggers when `FREELLMAPI_CONTEXT_HANDOFF=on_model_switch` and the selected model differs from `lastModelKey`, inserting a summary system message before provider-specific prompts.
- **Safety limits** include 500-character per-message truncation, 6,000-character summary caps, 500-session storage limits, and `HANDOFF_MAX_TOKENS` (~1,600) reserved in token budgets.
- **Integration points** are [`server/src/routes/proxy.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/routes/proxy.ts) for recording incoming messages and [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) for orchestrating the handoff check and model tracking.

## Frequently Asked Questions

### What triggers a context handoff in FreeLLMAPI?

A context handoff triggers when the `FREELLMAPI_CONTEXT_HANDOFF` environment variable is set to `"on_model_switch"` and the model selected for the current request differs from the `lastModelKey` stored in the session cache. The system detects this mismatch in `maybeInjectContextHandoff` and injects a summary system message so the new model understands the conversation history without receiving the full raw message log.

### How does FreeLLMAPI prevent token limit errors when injecting handoff messages?

The system reserves headroom using the exported constant `HANDOFF_MAX_TOKENS` (approximately 1,600 tokens). The router subtracts this value from the model's context window during token budget calculations before checking if the request fits. Additionally, the `buildSummary` function caps the generated summary at 6,000 characters, and individual messages are truncated to 500 characters before storage, keeping the injected payload well within typical model limits.

### Can I disable context handoff if I don't need model switching?

Yes. Set `FREELLMAPI_CONTEXT_HANDOFF=off` or leave the variable unset. When `getContextHandoffMode()` returns `'off'`, the `maybeInjectContextHandoff` function returns the original messages unchanged with `injected: false`, and the system performs no session storage or message injection, eliminating all associated overhead.