How Context Handoff Works When Switching Models Mid-Conversation in FreeLLMAPI

FreeLLMAPI preserves conversation continuity by injecting a summary system message when the environment variable FREELLMAPI_CONTEXT_HANDOFF is set to on_model_switch, comparing the last used model with the newly selected one and automatically briefing the new model on recent turns.

Maintaining context across different language models is challenging when routing requests dynamically. The FreeLLMAPI repository solves this through a server-side context handoff mechanism that activates when switching models mid-conversation, eliminating the need for client-side changes while preventing context window overflow.

The Three-Step Handoff Pipeline

The implementation in server/src/services/context-handoff.ts manages context preservation through three coordinated operations that execute during the request lifecycle.

Step 1: Recording Incoming Messages

When a chat request arrives, the proxy handler immediately calls recordIncomingMessages(sessionKey, messages) to update the session store. This function caps the history at the last 12 turns, truncates each message to 500 characters, and stores them with a timestamp. If the payload contains no assistant turns—indicating a fresh conversation—the function clears the lastModelKey field to reset the handoff state.

Step 2: Tracking the Last Successful Model

After a provider successfully returns a response, the router invokes recordSuccessfulModel({ sessionKey, modelKey }) to write the model identifier (e.g., "gpt-4" or "claude-2") into the session entry. This stored key serves as the comparison point for the next request to determine if a model switch occurred.

Step 3: Conditional Injection of Handoff Messages

Before routing a request to the selected provider, the system calls maybeInjectContextHandoff({ mode, sessionKey, messages, selectedModelKey }). When mode is 'on_model_switch' and the stored lastModelKey differs from the selectedModelKey, the function generates a summary via buildSummary of the stored recent messages (capped at 6,000 characters) and inserts a system message immediately after any leading system messages. The injected message follows this format:

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (<prev> → <new>).
... (instructions) ...
Recent session summary:
User: ...
Assistant: ...

The function returns injectedTokens, which the router adds to its token budget calculations using the exported constant HANDOFF_MAX_TOKENS (approximately 1,600 tokens).

Configuration and Activation

Context handoff is disabled by default. To enable the feature, set the environment variable before starting the server:

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

The getContextHandoffMode() function in server/src/services/context-handoff.ts reads this variable, trims and lowercases the value, and returns either 'on_model_switch' or 'off'. When set to 'off', the middleware skips all injection logic and passes requests through unchanged.

Integration Points in the Request Lifecycle

The handoff mechanism integrates at three critical points in server/src/routes/proxy.ts and server/src/services/router.ts:

1. Request ingestion: The proxy handler calls recordIncomingMessages as soon as the payload arrives, ensuring the session store reflects the latest user input regardless of which model will handle it.

2. Pre-routing: The router executes maybeInjectContextHandoff to check for model switches and potentially inject the summary message before forwarding to the provider.

3. Post-response: After receiving a successful provider response, the router calls recordSuccessfulModel to update the session with the model that just completed the turn.

Safety Mechanisms and Memory Limits

FreeLLMAPI implements several safeguards to prevent unbounded growth and token overflow:

• Session cap: The store limits active sessions to 500 entries, evicting the oldest after a 3-hour TTL.
• Deduplication: The injection logic detects if the provider already received a hand-off system message (e.g., manually added by the user) via the alreadyPresent check, preventing duplicates.
• Token budgeting: The constant HANDOFF_MAX_TOKENS reserves headroom in the routing token estimate, ensuring the injected summary never exceeds the model's context window.

Implementation Examples

Enabling Context Handoff

Set the environment variable in your .env file or runtime environment:

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

Recording User Turns

Import and call the recording function in your request handler:

import { recordIncomingMessages } from '@freellmapi/server/src/services/context-handoff';

// sessionKey derives from a cookie or Authorization header
recordIncomingMessages(sessionKey, incomingChatMessages);

Injecting Handoff Messages Before Routing

Check the mode and conditionally inject context when switching models:

import { maybeInjectContextHandoff, getContextHandoffMode } from '@freellmapi/server/src/services/context-handoff';

const mode = getContextHandoffMode();
const { messages, injected, injectedTokens } = maybeInjectContextHandoff({
  mode,
  sessionKey,
  messages: originalMessages,          // payload from the client
  selectedModelKey: providerModelKey,  // e.g., "gpt-4", "claude-2"
});

Tracking Model Success

After a successful response, update the session to remember which model handled the turn:

import { recordSuccessfulModel } from '@freellmapi/server/src/services/context-handoff';

recordSuccessfulModel({ sessionKey, modelKey: providerModelKey });

Reserving Token Budget

Use the exported constant to account for potential handoff overhead:

import { HANDOFF_MAX_TOKENS } from '@freellmapi/server/src/services/context-handoff';

const tokenBudget = modelContextWindow - requestTokens - HANDOFF_MAX_TOKENS;

Summary

• FreeLLMAPI enables seamless context handoff when switching models mid-conversation through the FREELLMAPI_CONTEXT_HANDOFF environment variable set to on_model_switch.
• The system stores the last 12 turns (500 characters each) in server/src/services/context-handoff.ts, tracking which model last responded via recordSuccessfulModel.
• When models differ, maybeInjectContextHandoff automatically inserts a summary system message capped at 6,000 characters and approximately 1,600 tokens.
• Integration occurs in server/src/routes/proxy.ts (recording input) and server/src/services/router.ts (injection and tracking).
• Safety mechanisms include a 500-session limit with 3-hour TTL, deduplication checks, and token budget reservation via HANDOFF_MAX_TOKENS.

Frequently Asked Questions

How does FreeLLMAPI detect when to trigger a context handoff?

FreeLLMAPI compares the lastModelKey stored in the session with the selectedModelKey of the incoming request. If the environment variable FREELLMAPI_CONTEXT_HANDOFF is set to on_model_switch and these keys differ, the maybeInjectContextHandoff function automatically generates a summary message. This detection happens server-side in the router before forwarding the request to the new provider.

What happens if context handoff is disabled?

When FREELLMAPI_CONTEXT_HANDOFF is set to off or undefined, getContextHandoffMode() returns 'off', causing maybeInjectContextHandoff to skip all injection logic. The request passes through unchanged, meaning the new model receives no summary of previous turns, effectively starting a fresh conversation from that model's perspective.

How does the system prevent duplicate handoff messages?

The maybeInjectContextHandoff function checks for an alreadyPresent flag that detects if the messages array already contains a handoff system message (such as one manually added by the client). If detected, the function avoids injecting a duplicate summary. Additionally, the system inserts the handoff message immediately after any existing system messages to preserve provider-specific prompt ordering.

What are the memory and token limits for stored context?

The session store caps history at 12 turns with 500 characters per message, and the generated summary is limited to 6,000 characters. The system maintains a maximum of 500 active sessions with a 3-hour TTL to prevent memory leaks. Token budgeting reserves approximately 1,600 tokens via HANDOFF_MAX_TOKENS to ensure the injected summary never exceeds the model's context window.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}