How FreeLLMAPI Implements Context Handoff When Switching Models Mid-Session

FreeLLMAPI maintains continuity across model switches by storing recent conversation turns in a lightweight session cache and automatically injecting a contextual summary system message when a different model takes over the next turn.

FreeLLMAPI is an open-source proxy that routes LLM requests to multiple providers, and it solves the critical problem of context handoff when users switch between models mid-conversation. Rather than forcing clients to manage state or resend entire histories, the server intercepts model switches and synthesizes a handoff message that compresses recent context for the new model.

The Three-Step Handoff Mechanism

The context handoff implementation lives primarily in server/src/services/context-handoff.ts and operates through three coordinated stages that track conversation state and trigger summary injection.

Recording Incoming Messages

When a request first enters the system, the proxy handler immediately calls recordIncomingMessages(sessionKey, messages) to update the session store. This function trims the payload to the last 12 turns and limits each message to 500 characters, storing them with a timestamp. If the payload contains no assistant turns—indicating a fresh conversation—it clears the lastModelKey field to reset the handoff state.

import { recordIncomingMessages } from '@freellmapi/server/src/services/context-handoff';

// Called at the start of every request in server/src/routes/proxy.ts
recordIncomingMessages(sessionSessionKey, incomingChatMessages);

The session store enforces hard limits: it caps at 500 sessions and evicts entries older than 3 hours, ensuring the cache never grows unbounded.

Tracking the Last Successful Model

After a provider successfully returns a response, the router calls recordSuccessfulModel({ sessionKey, modelKey }) to record which model handled the turn. This writes the model key (e.g., "gpt-4" or "claude-2") into the session entry, creating a breadcrumb that the next request can compare against.

import { recordSuccessfulModel } from '@freellmapi/server/src/services/context-handoff';

// Called after successful provider response in server/src/services/router.ts
recordSuccessfulModel({ sessionKey, modelKey: providerModelKey });

Injecting the Handoff Summary

Before routing a request to the selected provider, the system invokes maybeInjectContextHandoff({ mode, sessionKey, messages, selectedModelKey }). When the environment variable FREELLMAPI_CONTEXT_HANDOFF is set to "on_model_switch" and the stored lastModelKey differs from the newly selected model, the function builds a conversation summary using buildSummary()—capped at 6,000 characters—and inserts a system message immediately after any existing system prompts.

The injected message follows this structure:

FreeLLMAPI context handoff:
You are taking over an ongoing conversation from another model (<previous_model> → <new_model>).
... (instructions) ...
Recent session summary:
User: ...
Assistant: ...

The function returns injectedTokens (estimated using the exported constant HANDOFF_MAX_TOKENS, roughly 1,600 tokens), which the router adds to its token budget calculations to prevent context window overflows.

import { maybeInjectContextHandoff, getContextHandoffMode } from '@freellmapi/server/src/services/context-handoff';

const mode = getContextHandoffMode(); // Reads FREELLMAPI_CONTEXT_HANDOFF env var
const { messages, injected, injectedTokens } = maybeInjectContextHandoff({
  mode,
  sessionKey,
  messages: originalMessages,
  selectedModelKey: providerModelKey,
});

Configuration and Environment Setup

The entire handoff mechanism is gated by the FREELLMAPI_CONTEXT_HANDOFF environment variable. The function getContextHandoffMode() in server/src/services/context-handoff.ts reads this variable and returns either 'on_model_switch' or 'off'. When set to 'off', the system skips injection entirely and passes requests through unchanged.

# Enable automatic context handoff on model switches

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

This design keeps the feature opt-in and ensures zero overhead when disabled.

Integration with the Request Lifecycle

The handoff logic integrates at two critical points in the request pipeline to maintain stateless operation while preserving context across turns.

Proxy Handler Initialization

In server/src/routes/proxy.ts, the incoming request handler immediately records the user's messages via recordIncomingMessages. This ensures the session cache stays current regardless of which model ultimately processes the request or whether the request succeeds.

Router Orchestration

The server/src/services/router.ts file orchestrates the handoff check. Before forwarding to the selected provider, it calls maybeInjectContextHandoff to potentially prepend the summary. After receiving a successful response, it updates the model tracking via recordSuccessfulModel. This creates a closed loop where every turn updates the session state for the next.

Safety Mechanisms and Limits

FreeLLMAPI implements several safeguards to ensure context handoff never degrades performance or breaks provider limits:

• Token Budget Safety: The exported constant HANDOFF_MAX_TOKENS (~1,600 tokens) is subtracted from the model's available context window during routing calculations, guaranteeing the injected summary never pushes requests over the limit.
• Duplicate Prevention: The injection logic checks for existing handoff messages (alreadyPresent) to avoid duplicating summaries if a user manually includes one or if the same request is retried.
• Bounded Storage: The session store's 500-entry cap and 3-hour TTL prevent memory leaks in long-running server instances.
• Graceful Degradation: If the session store is unavailable or the summary generation fails, the system falls back to sending the original messages without injection, preserving availability over context continuity.

import { HANDOFF_MAX_TOKENS } from '@freellmapi/server/src/services/context-handoff';

// Reserve headroom for potential handoff injection
const tokenBudget = modelContextWindow - requestTokens - HANDOFF_MAX_TOKENS;

Summary

• FreeLLMAPI implements context handoff through a lightweight session cache in server/src/services/context-handoff.ts that stores the last 12 conversation turns.
• Automatic injection triggers when FREELLMAPI_CONTEXT_HANDOFF=on_model_switch and the selected model differs from lastModelKey, inserting a summary system message before provider-specific prompts.
• Safety limits include 500-character per-message truncation, 6,000-character summary caps, 500-session storage limits, and HANDOFF_MAX_TOKENS (~1,600) reserved in token budgets.
• Integration points are server/src/routes/proxy.ts for recording incoming messages and server/src/services/router.ts for orchestrating the handoff check and model tracking.

Frequently Asked Questions

What triggers a context handoff in FreeLLMAPI?

A context handoff triggers when the FREELLMAPI_CONTEXT_HANDOFF environment variable is set to "on_model_switch" and the model selected for the current request differs from the lastModelKey stored in the session cache. The system detects this mismatch in maybeInjectContextHandoff and injects a summary system message so the new model understands the conversation history without receiving the full raw message log.

How does FreeLLMAPI prevent token limit errors when injecting handoff messages?

The system reserves headroom using the exported constant HANDOFF_MAX_TOKENS (approximately 1,600 tokens). The router subtracts this value from the model's context window during token budget calculations before checking if the request fits. Additionally, the buildSummary function caps the generated summary at 6,000 characters, and individual messages are truncated to 500 characters before storage, keeping the injected payload well within typical model limits.

Can I disable context handoff if I don't need model switching?

Yes. Set FREELLMAPI_CONTEXT_HANDOFF=off or leave the variable unset. When getContextHandoffMode() returns 'off', the maybeInjectContextHandoff function returns the original messages unchanged with injected: false, and the system performs no session storage or message injection, eliminating all associated overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}