How Context Handoff Works in FreeLLMAPI: Preserving Conversation Continuity Across Model Switches

FreeLLMAPI preserves conversation continuity when switching models by injecting a synthesized system message containing a truncated session summary, ensuring the new model receives critical context from previous turns without consuming excessive tokens.

FreeLLMAPI is an open-source proxy that unifies multiple large language model providers under a single API. When users switch between models mid-conversation, the context handoff mechanism ensures the new model understands the conversation history without requiring clients to manually resend entire message threads.

How Context Handoff Works in FreeLLMAPI

The context handoff system operates through six distinct phases implemented in server/src/services/context-handoff.ts. Each phase ensures that conversation state persists even when routing changes the underlying model provider.

Tracking Session Messages

Every inbound request records the last approximately 12 messages (truncated to 500 characters each) in an in-memory Map keyed by the client-provided sessionKey. This logic resides in the recordIncomingMessages function (lines 56-79), which maintains a rolling buffer of conversation context accessible across different model calls.

Recording the Last Successful Model

After a model completes a turn successfully, the service stores its identifier using the format platform:modelId in the same session entry. The recordSuccessfulModel function (lines 59-73) updates the lastModelKey property, creating a breadcrumb trail of which model handled the previous interaction.

Detecting Model Switches

On subsequent requests, the proxy compares the stored lastModelKey against the model selected for the current turn. The maybeInjectContextHandoff function (lines 111-154) performs this comparison and determines whether a handoff is necessary based on the FREELLMAPI_CONTEXT_HANDOFF environment variable.

Building a Concise Summary

When a switch is detected, the service invokes buildSummary (lines 94-100) to concatenate stored messages into a condensed narrative. This summary is capped at 6000 characters to balance context richness against token economy.

Injecting the System-Message Handoff

The summary is wrapped in a system-role message beginning with FreeLLMAPI context handoff: and inserted after any existing system prompts in maybeInjectContextHandoff (lines 134-152). This positioning ensures provider-native system prompt ordering remains intact while giving the new model immediate awareness of the conversation trajectory.

Token Head-Room Management

To prevent context window overflows, the proxy inflates routing token estimates by HANDOFF_MAX_TOKENS (approximately 1600 tokens) whenever a handoff might be injected. This padding, defined in server/src/services/context-handoff.ts (lines 21-24) and applied in server/src/routes/proxy.ts (lines 54-60), reserves capacity for the injected system message.

Configuration and Environment Variables

The entire flow is gated by the FREELLMAPI_CONTEXT_HANDOFF environment variable. When set to on_model_switch, handoff logic activates in getContextHandoffMode (lines 27-30). Any other value, or an unset variable, disables the feature entirely.

# .env or exported before starting the server

export FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

Implementation Examples

Initial Request with Session Tracking

Clients must provide a stable sessionKey to participate in handoff tracking. This identifier acts as the primary key for the in-memory message store.

import axios from 'axios';

// The `sessionKey` is a stable identifier for a user conversation (e.g. a UUID).
const payload = {
  model: 'gpt-4o',          // first turn – model A
  messages: [{ role: 'user', content: 'Explain quantum tunnelling.' }],
  sessionKey: '123e4567-e89b-12d3-a456-426614174000',
};

await axios.post('https://api.freellmapi.com/v1/chat/completions', payload);

Switching Models Mid-Conversation

When submitting a follow-up request to a different model, include the same sessionKey. FreeLLMAPI detects the switch and injects the context handoff automatically.

const secondPayload = {
  model: 'anthropic/claude-3',  // model B – different platform
  messages: [
    { role: 'assistant', content: '... (previous answer)' },
    { role: 'user', content: 'Now apply that to a semiconductor device.' },
  ],
  sessionKey: '123e4567-e89b-12d3-a456-426614174000',
};

await axios.post('https://api.freellmapi.com/v1/chat/completions', secondPayload);

When processing the second request, FreeLLMAPI detects that lastModelKey (gpt-4o) differs from the selected model (anthropic/claude-3). The proxy injects a system message similar to:

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (gpt-4o → anthropic/claude-3).
Continue the user's task using the conversation context already provided in this request.
Do not restart the task, re‑ask already answered setup questions, or discard prior tool results.
Respect the user's latest message as the highest‑priority instruction.

Recent session summary:
User: Explain quantum tunnelling.
Assistant: ... (truncated)
User: Now apply that to a semiconductor device.

Debugging Session Storage

For testing purposes, you can inspect or clear the internal store using the testing utility.

import { _clearStoreForTesting } from './services/context-handoff.js';

// In a test environment you can clear the map:
_clearStoreForTesting();

Key Implementation Files

The context handoff system spans several files in the FreeLLMAPI repository:

• server/src/services/context-handoff.ts: Core logic for mode handling, message trimming, store management, injection, and token accounting.
• server/src/routes/proxy.ts: Where the handoff is optionally injected into outgoing requests and where token padding (HANDOFF_MAX_TOKENS) is applied.
• server/src/services/request-retention.ts: Provides the sessionKey plumbing used by recordIncomingMessages.
• server/src/services/model-listing.ts: Supplies the model-selection pipeline that yields the selectedModelKey used for handoff detection.

Summary

• Context handoff activates when FREELLMAPI_CONTEXT_HANDOFF is set to on_model_switch and the lastModelKey differs from the current selection.
• The system stores approximately 12 recent messages per sessionKey in an in-memory Map, truncating each to 500 characters.
• maybeInjectContextHandoff in server/src/services/context-handoff.ts handles detection and injection logic (lines 111-154).
• A 6000-character summary caps the injected context, with HANDOFF_MAX_TOKENS (1600 tokens) reserved in the token budget.
• The handoff message is positioned after existing system prompts to respect provider-native ordering.

Frequently Asked Questions

What triggers a context handoff in FreeLLMAPI?

A context handoff triggers when two conditions align: the FREELLMAPI_CONTEXT_HANDOFF environment variable is set to on_model_switch, and the model selected for the current request differs from the lastModelKey stored in the session Map. The maybeInjectContextHandoff function performs this detection in server/src/services/context-handoff.ts (lines 111-154).

How does FreeLLMAPI manage token limits during context handoff?

The proxy preemptively allocates HANDOFF_MAX_TOKENS (approximately 1600 tokens) to the routing budget whenever handoff is possible, as implemented in server/src/routes/proxy.ts (lines 54-60). Additionally, the buildSummary function limits the injected summary to 6000 characters, ensuring the downstream model's context window remains sufficient for the actual response generation.

Can I disable context handoff if my application doesn't switch models?

Yes. Set the FREELLMAPI_CONTEXT_HANDOFF environment variable to any value other than on_model_switch, or leave it unset entirely. The getContextHandoffMode function (lines 27-30 in server/src/services/context-handoff.ts) returns the active mode, and any non-matching value disables the injection logic entirely while maintaining normal proxy functionality.

Does context handoff work across different LLM providers?

Yes. The mechanism is provider-agnostic. The lastModelKey stores identifiers in platform:modelId format (for example, openai:gpt-4o or anthropic:claude-3), and the handoff summary is injected as a standard system message compatible with OpenAI, Anthropic, and other supported API formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}