How FreeLLMAPI Context Handoff Preserves Conversations When Switching Models

FreeLLMAPI preserves conversation continuity across model switches by injecting a compact session summary as a system message whenever the selected model differs from the one that handled the previous turn.

FreeLLMAPI, an open-source LLM proxy from tashfeenahmed/freellmapi, solves a common multi-model routing problem: when a user switches from one model to another mid-conversation, the new model lacks the implicit context of prior turns. The context handoff feature bridges this gap without exposing provider-specific state management to the client.

Enabling the Context Handoff Feature

The entire flow is gated by the FREELLMAPI_CONTEXT_HANDOFF environment variable. When set to on_model_switch, FreeLLMAPI evaluates every request for a potential handoff. Any other value, or an unset variable, completely disables the logic.

According to the source in server/src/services/context-handoff.ts, the function getContextHandoffMode reads this variable and returns the active mode (lines 27-30). The proxy only proceeds with injection if this mode matches the required trigger.

# .env or exported before starting the server

export FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

How the Context Handoff Mechanism Works

The core implementation lives in server/src/services/context-handoff.ts. It operates as a six-step pipeline that runs per request.

1. Tracking Recent Messages Per Session

Every inbound request records the last ~12 messages in an in-memory Map keyed by the client-provided sessionKey. The helper recordIncomingMessages trims each message to roughly 500 characters before storage to bound memory growth (lines 56-79).

Clients must supply a stable sessionKey (for example, a UUID) across turns. Without this key, the service cannot correlate turns and the handoff window is lost.

2. Remembering the Last Successful Model

After a model finishes a turn successfully, recordSuccessfulModel stores its identifier as a composite lastModelKey string—formatted as platform:modelId—inside the same session entry (lines 59-73). This stored key becomes the baseline for comparison on the next request.

3. Detecting a Model Switch

On each subsequent request, maybeInjectContextHandoff compares the stored lastModelKey against the newly chosen selectedModelKey (lines 111-154). If the keys differ, the service flags a handoff event and proceeds to summary generation.

4. Building a Session Summary

If a switch is detected, buildSummary concatenates the stored, trimmed messages into a single string capped at 6000 characters (lines 94-100). This truncated history is deliberately compact to avoid consuming excessive context-window space.

5. Injecting the System-Message Handoff

The summary is wrapped in a system-role message prefixed with FreeLLMAPI context handoff:. The proxy inserts this message after any leading system prompts already present in the request, preserving the provider’s native system prompt ordering (lines 134-152).

The injected message instructs the new model to continue the user's task without restarting, re-asking setup questions, or discarding prior tool results.

6. Accounting for Token Headroom

To prevent downstream context-window overflows, the proxy inflates the routing token estimate by HANDOFF_MAX_TOKENS (approximately 1600 tokens) whenever a handoff could be injected. This padding is applied in server/src/routes/proxy.ts (lines 54-60) before the request reaches the target provider.

Practical Example: Switching from GPT-4o to Claude

The following pattern demonstrates a conversation that begins on OpenAI and continues on Anthropic.

First, the client initiates a turn using a specific sessionKey:

import axios from 'axios';

// The `sessionKey` is a stable identifier for a user conversation (e.g. a UUID).
const payload = {
  model: 'gpt-4o',          // first turn – model A
  messages: [{ role: 'user', content: 'Explain quantum tunnelling.' }],
  sessionKey: '123e4567-e89b-12d3-a456-426614174000',
};

await axios.post('https://api.freellmapi.com/v1/chat/completions', payload);

On the next turn, the client requests a different model while keeping the same sessionKey:

const secondPayload = {
  model: 'anthropic/claude-3',  // model B – different platform
  messages: [
    { role: 'assistant', content: '... (previous answer)' },
    { role: 'user', content: 'Now apply that to a semiconductor device.' },
  ],
  sessionKey: '123e4567-e89b-12d3-a456-426614174000',
};

await axios.post('https://api.freellmapi.com/v1/chat/completions', secondPayload);

Because lastModelKey (gpt-4o) differs from the selected model (anthropic/claude-3), FreeLLMAPI injects a system message similar to:

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (gpt-4o → anthropic/claude-3).
Continue the user's task using the conversation context already provided in this request.
Do not restart the task, re‑ask already answered setup questions, or discard prior tool results.
Respect the user's latest message as the highest‑priority instruction.

Recent session summary:
User: Explain quantum tunnelling.
Assistant: ... (truncated)
User: Now apply that to a semiconductor device.

The downstream Claude model receives this preamble first, allowing it to continue seamlessly.

Key Source Files and Responsibilities

Several modules collaborate to make the context handoff feature reliable:

• server/src/services/context-handoff.ts — Contains the core logic, including mode handling (getContextHandoffMode), message trimming (recordIncomingMessages), success tracking (recordSuccessfulModel), summary generation (buildSummary), and injection (maybeInjectContextHandoff).
• server/src/routes/proxy.ts — Orchestrates the optional injection and applies the HANDOFF_MAX_TOKENS padding to token estimates (lines 54-60).
• server/src/services/request-retention.ts — Provides the sessionKey plumbing that recordIncomingMessages depends on to key the in-memory store.
• server/src/services/model-listing.ts — Supplies the model-selection pipeline that yields the selectedModelKey used for switch detection.

Summary

• FreeLLMAPI context handoff preserves multi-turn conversations when routing switches to a different model mid-session.
• The feature requires FREELLMAPI_CONTEXT_HANDOFF=on_model_switch and a stable client-provided sessionKey.
• recordIncomingMessages tracks the last ~12 truncated messages in-memory per session, while recordSuccessfulModel remembers the prior model identifier.
• maybeInjectContextHandoff detects model switches and injects a compact system summary after existing system prompts.
• The proxy reserves HANDOFF_MAX_TOKENS (~1600 tokens) in server/src/routes/proxy.ts to protect downstream context-window limits.

Frequently Asked Questions

What triggers a context handoff in FreeLLMAPI?

A context handoff triggers when two conditions are met: the environment variable FREELLMAPI_CONTEXT_HANDOFF is set to on_model_switch, and the lastModelKey stored for the current sessionKey differs from the newly selected model's platform:modelId composite key. The check happens inside maybeInjectContextHandoff in server/src/services/context-handoff.ts (lines 111-154).

How does the context handoff feature handle token limits?

The service caps the generated session summary at 6000 characters during buildSummary and preemptively inflates the routing token estimate by HANDOFF_MAX_TOKENS (approximately 1600 tokens) in server/src/routes/proxy.ts (lines 54-60). This padding ensures the downstream provider's context-window validation accounts for the extra system message before the request is forwarded.

Can I disable context handoff entirely?

You can disable the feature by unsetting the FREELLMAPI_CONTEXT_HANDOFF environment variable or by setting it to any value other than on_model_switch. The helper getContextHandoffMode in server/src/services/context-handoff.ts (lines 27-30) evaluates this variable, and the proxy skips all injection logic when the mode does not match.

Why is the sessionKey parameter required for handoff?

The in-memory store is a Map keyed by sessionKey. recordIncomingMessages uses this key to correlate turns and accumulate conversation history (lines 56-79). Without a stable sessionKey across requests, FreeLLMAPI cannot retrieve the prior model identifier or message window, so no handoff summary can be generated.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}