How FreeLLMAPI Manages Context Handoff When Switching Models Mid-Conversation

FreeLLMAPI maintains a lightweight per-session store that records recent conversation turns and the previous model identifier, automatically injecting a summary system message when FREELLMAPI_CONTEXT_HANDOFF=on_model_switch detects a provider change.

FreeLLMAPI is an open-source LLM gateway that enables dynamic routing between multiple AI providers without losing conversational continuity. When users switch models mid-conversation—transitioning from GPT-4 to Claude or any other supported backend—the system must ensure the new model comprehends the prior context without requiring client-side state management. The repository solves this through a server-side context handoff mechanism implemented in server/src/services/context-handoff.ts.

The Context Handoff Architecture

The implementation relies on a stateless API surface with bounded server-side storage. Rather than forcing clients to resubmit entire conversation histories or maintain complex state, FreeLLMAPI tracks three critical pieces of data per session: the last 12 conversation turns, the timestamp of the last activity, and the identifier of the model that processed the previous request. This design ensures that context handoff when switching models mid-conversation happens transparently, with the injected system message providing the new model with a concise summary of the discussion so far.

Step-by-Step Implementation

The context handoff logic proceeds through three distinct phases, each handled by specific functions in the handoff service.

1. Recording Incoming Messages

When a request arrives at the proxy handler (server/src/routes/proxy.ts), the system immediately calls recordIncomingMessages(sessionKey, messages) to update the session store. This function performs aggressive trimming to control memory usage:

• Retains only the last 12 turns of the conversation
• Limits each message to 500 characters
• Clears the lastModelKey field if the payload contains no assistant turns (indicating a fresh conversation)

This ensures the store never grows unbounded while preserving the most relevant recent context.

2. Tracking the Active Model

After the router (server/src/services/router.ts) successfully receives a response from the selected provider, it invokes recordSuccessfulModel({sessionKey, modelKey}). This writes the provider identifier (e.g., "gpt-4" or "claude-2") into the session entry, enabling the next request to detect whether a different model has been selected.

3. Injecting the Handoff Message

Before routing a request, the system calls maybeInjectContextHandoff({mode, sessionKey, messages, selectedModelKey}). If the environment variable FREELLMAPI_CONTEXT_HANDOFF is set to "on_model_switch" and the stored lastModelKey differs from the newly selected model, the function:

1. Generates a summary via buildSummary(), capped at 6,000 characters
2. Constructs a system message with the template: FreeLLMAPI context handoff: You are taking over an ongoing conversation from another model (<prev> → <new>)...
3. Inserts the message immediately after any existing system messages, preserving provider-specific prompt ordering
4. Returns the estimated token count (injectedTokens) alongside the modified message array

The function also checks for duplicate handoff messages (alreadyPresent) to prevent redundant injection if the user manually included a similar instruction.

Configuration and Environment Variables

The entire handoff mechanism is gated by the getContextHandoffMode() function, which reads the environment at runtime:

export function getContextHandoffMode(): ContextHandoffMode {
  const raw = process.env.FREELLMAPI_CONTEXT_HANDOFF?.trim().toLowerCase();
  return raw === 'on_model_switch' ? 'on_model_switch' : 'off';
}

When the mode is "off" (the default), the system skips injection entirely and passes requests through unchanged. To enable automatic context handoff when switching models mid-conversation, set:

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

Integration Points in the Request Lifecycle

The handoff service integrates at three critical points in the request pipeline:

• Entry point: The proxy handler (server/src/routes/proxy.ts) calls recordIncomingMessages immediately upon receiving the user payload, ensuring the store stays current regardless of routing decisions
• Pre-routing: The router (server/src/services/router.ts) invokes maybeInjectContextHandoff before forwarding requests to the selected provider, allowing the injected summary to reach the new model
• Post-response: The router calls recordSuccessfulModel after receiving a successful provider response, updating the session with the model that just completed the turn

Safety Mechanisms and Token Budgeting

FreeLLMAPI implements several safeguards to prevent the handoff mechanism from overwhelming models or exhausting server resources:

• Token budgeting: The exported constant HANDOFF_MAX_TOKENS (approximately 1,600 tokens) is added to the router's token estimate, ensuring the injected summary never pushes the request beyond the model's context window
• Memory bounds: The session store caps at 500 active sessions and evicts entries after a 3-hour TTL, preventing unbounded growth
• Deduplication: The injection logic detects existing handoff messages to avoid duplicate context summaries

Code Examples

Enabling Context Handoff

Set the environment variable before starting the server:

# .env

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

Recording User Turns

Call this when handling incoming chat requests:

import { recordIncomingMessages } from '@freellmapi/server/src/services/context-handoff';

// sessionKey derived from cookie or Authorization header
recordIncomingMessages(sessionKey, incomingChatMessages);

Checking and Injecting Context

Use this in your request router before forwarding to the provider:

import { maybeInjectContextHandoff, getContextHandoffMode } from '@freellmapi/server/src/services/context-handoff';

const mode = getContextHandoffMode();
const { messages, injected, injectedTokens } = maybeInjectContextHandoff({
  mode,
  sessionKey,
  messages: originalMessages,
  selectedModelKey: providerModelKey, // e.g., "gpt-4", "claude-2"
});

Updating the Model Tracker

Call after receiving a successful response:

import { recordSuccessfulModel } from '@freellmapi/server/src/services/context-handoff';

recordSuccessfulModel({ sessionKey, modelKey: providerModelKey });

Reserving Token Budget

Account for handoff overhead in your token calculations:

import { HANDOFF_MAX_TOKENS } from '@freellmapi/server/src/services/context-handoff';

const tokenBudget = modelContextWindow - requestTokens - HANDOFF_MAX_TOKENS;

Summary

• FreeLLMAPI manages context handoff when switching models mid-conversation through a server-side session store that tracks the last 12 turns and the previous model identifier
• The recordIncomingMessages function in server/src/services/context-handoff.ts maintains the conversation buffer with strict limits (12 turns, 500 chars per message)
• maybeInjectContextHandoff automatically inserts a summary system message when it detects a model change and the FREELLMAPI_CONTEXT_HANDOFF environment variable is set to "on_model_switch"
• The HANDOFF_MAX_TOKENS constant (~1,600 tokens) ensures the injected summary never exceeds the model's context window
• The store implements TTL-based eviction (3 hours) and caps at 500 sessions to prevent memory leaks

Frequently Asked Questions

What happens if context handoff is disabled?

When FREELLMAPI_CONTEXT_HANDOFF is unset or set to "off", the getContextHandoffMode() function returns "off" and maybeInjectContextHandoff returns the original messages unchanged. The system passes requests directly to the new model without injecting summary context, meaning the model receives only the conversation history explicitly sent by the client.

How does FreeLLMAPI prevent the handoff message from exceeding token limits?

The system uses the exported constant HANDOFF_MAX_TOKENS (approximately 1,600 tokens) as a safety margin. The router subtracts this value from the available context window before checking if the request fits within the model's limits. Additionally, the buildSummary function caps the generated summary at 6,000 characters, and individual stored messages are truncated to 500 characters each.

Can the context handoff message template be customized?

Currently, the handoff message template is hardcoded within the maybeInjectContextHandoff function in server/src/services/context-handoff.ts. The template includes the previous and new model names, specific instructions for the incoming model, and the recent session summary. Users cannot modify this template via configuration files; customization would require editing the source code and rebuilding the service.

Where is the conversation history stored during a session?

The conversation history resides in a server-side in-memory store maintained by the context handoff service. This store caps at 500 concurrent sessions and automatically evicts entries after 3 hours of inactivity. The client never needs to maintain this state, enabling truly stateless API interactions while preserving continuity across model switches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}