How FreeLLMAPI Handles Context Handoff When Switching Models Mid-Conversation

FreeLLMAPI automatically transfers recent conversation history to new models during mid-conversation switches by injecting a system handoff message containing a summarized context window up to 6,000 characters.

FreeLLMAPI's intelligent routing layer can transparently shift traffic between different LLM providers when rate limits hit or better models become available. To maintain conversational continuity during these transitions, the platform implements a context handoff mechanism that preserves up to 12 recent message turns and injects them as a system prompt when FREELLMAPI_CONTEXT_HANDOFF is set to on_model_switch. This process is fully implemented in server/src/services/context-handoff.ts.

Enabling Context Handoff

The handoff system is controlled via environment configuration. By default, the feature is disabled to minimize token usage.

Configuration Mode

The getContextHandoffMode() function reads the FREELLMAPI_CONTEXT_HANDOFF environment variable (lines 27-30) to determine behavior:

• off: Disables context handoff entirely
• on_model_switch: Activates automatic context injection when the routing layer selects a different model than the previous turn

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

When enabled, the router reserves additional token headroom using the HANDOFF_MAX_TOKENS constant (lines 21-24), approximately 1,500 tokens, to accommodate the injected summary.

The Context Handoff Workflow

The handoff process operates across multiple request cycles, tracking session state in an in-memory Map with a 3-hour TTL (SESSION_TTL_MS).

Recording Incoming Messages

Before routing each request, recordIncomingMessages() (lines 56-78) persists the conversation history. It stores the last 12 user and assistant messages, trimming each to 500 characters maximum. This data is keyed by the session identifier and maintained in an in-memory store limited to 500 entries (MAX_STORE_SIZE).

import { recordIncomingMessages } from './services/context-handoff';

recordIncomingMessages(sessionKey, [
  { role: 'user', content: 'What is the weather in London?' },
  { role: 'assistant', content: 'It is 12°C and partly cloudy.' },
]);

Tracking Model Selection

After a provider successfully returns a response, recordSuccessfulModel() (lines 59-71) stores the lastModelKey that served the request. This enables the system to detect when the next request routes to a different provider.

import { recordSuccessfulModel } from './services/context-handoff';

recordSuccessfulModel(sessionKey, 'gpt-4');

Injecting Context on Model Switches

The maybeInjectContextHandoff() function (lines 111-156) implements the core injection logic. When invoked by the router (server/src/services/router.ts), it performs the following validation:

1. Verifies the mode is on_model_switch and a sessionKey exists
2. Retrieves the stored SessionContext and compares lastModelKey against the newly selected model
3. Guards against duplicate handoff messages
4. Generates a summary limited to 6,000 characters via buildSummary() (lines 94-100)
5. Inserts a system-role handoff message immediately after existing system prompts

import { maybeInjectContextHandoff, getContextHandoffMode } from './services/context-handoff';

const mode = getContextHandoffMode();
const { messages, injected, injectedTokens } = maybeInjectContextHandoff({
  mode,
  sessionKey,
  messages: incomingMessages,
  selectedModelKey: 'claude-2',
});

If injected returns true, the messages array now contains the handoff system message.

The Handoff Message Format

When a switch is detected (e.g., from gpt-4 to claude-2), FreeLLMAPI constructs a system message that instructs the new model to continue the conversation without restarting:

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (gpt-4 → claude-2).
Continue the user's task using the conversation context already provided in this request.
Do not restart the task, re-ask already answered setup questions, or discard prior tool results.
Respect the user's latest message as the highest-priority instruction.

Recent session summary:
User: How do I reset my password?
Assistant: You can reset it via the Settings page…
...

The summary is assembled by concatenating the stored trimmed messages and truncating to the MAX_HANDOFF_CHARS limit.

Token Budget and Storage Constraints

The system implements strict limits to prevent context overflow:

• Message Retention: Maximum 12 recent turns per session
• Character Limit: 500 characters per message, 6,000 characters total for the handoff summary
• Token Reservation: HANDOFF_MAX_TOKENS constant (≈1,500 tokens) reserved by the router for handoff injection
• Storage TTL: Sessions expire after 3 hours (SESSION_TTL_MS)
• Capacity: In-memory store limited to 500 entries (MAX_STORE_SIZE)

The hasPriorModel() helper (lines 102-109) allows the router to check if a session has previous model history, enabling proactive token headroom padding before the injection step.

import { HANDOFF_MAX_TOKENS } from './services/context-handoff';

const tokenBudget = estimateTokens(messages) + (injected ? HANDOFF_MAX_TOKENS : 0);

Summary

• Context handoff maintains conversation continuity when FreeLLMAPI switches models mid-conversation by injecting a system summary message
• The feature is controlled via FREELLMAPI_CONTEXT_HANDOFF environment variable with modes off or on_model_switch
• Up to 12 recent messages are stored in-memory with a 3-hour TTL, trimmed to 500 characters each
• The maybeInjectContextHandoff() function in server/src/services/context-handoff.ts handles detection and injection when lastModelKey differs from the current selection
• Handoff summaries are capped at 6,000 characters (≈1,500 tokens) and inserted after existing system prompts

Frequently Asked Questions

What triggers a context handoff in FreeLLMAPI?

A context handoff triggers when the routing layer selects a different model than the one that handled the previous request in the same session, provided FREELLMAPI_CONTEXT_HANDOFF is set to on_model_switch. The system compares the stored lastModelKey against the newly selected model key to detect the transition.

How much conversation history does FreeLLMAPI preserve during a handoff?

FreeLLMAPI preserves up to 12 recent user and assistant messages per session. Each message is trimmed to 500 characters maximum, and the final handoff summary is limited to 6,000 characters (approximately 1,500 tokens). This balances context continuity with token economy.

Is context handoff enabled by default?

No, context handoff is disabled by default. You must explicitly set the environment variable FREELLMAPI_CONTEXT_HANDOFF=on_model_switch to enable the feature. When disabled, model switches occur without injecting historical context, and the new model sees only the current request messages.

How does FreeLLMAPI prevent duplicate handoff messages?

The maybeInjectContextHandoff() function includes deduplication logic that guards against injecting multiple handoff messages for the same conversation turn. It checks the existing message array to ensure a handoff system message has not already been inserted before adding the context summary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}