What Is FREELLMAPI_CONTEXT_HANDOFF? Context Preservation Across Model Switches in FreeLLM

The FREELLMAPI_CONTEXT_HANDOFF environment variable enables context hand-off, a mechanism that preserves conversation history when routing requests between different LLM models by automatically injecting a system message containing a summary of recent turns.

In multi-model LLM deployments, requests often bounce between different providers or model versions based on availability, cost, or capability. Without intervention, each new model starts with zero context, forcing users to repeat information. The FREELLMAPI_CONTEXT_HANDOFF feature in the tashfeenahmed/freellmapi repository solves this by maintaining an in-memory record of recent conversations and seamlessly briefing new models when a switch occurs.

How Context Hand-Off Works

When FREELLMAPI_CONTEXT_HANDOFF is set to on_model_switch, the server activates a three-stage pipeline that runs during request processing.

Tracking Conversation History

The system records every incoming request to build a short-term memory buffer. The recordIncomingMessages function in server/src/services/context-handoff.ts stores recent user and assistant turns in an in-memory dictionary keyed by the session identifier. To prevent unbounded memory growth, the history is trimmed to a safe size before storage.

// Located at server/src/services/context-handoff.ts (lines 56-78)
recordIncomingMessages(sessionKey, messages);

Simultaneously, after a provider returns a successful response, recordSuccessfulModel captures the model key that generated the reply. This creates a breadcrumb trail showing which model last handled the session.

// Located at server/src/services/context-handoff.ts (lines 59-71)
recordSuccessfulModel({
  sessionKey: req.headers['x-session-id'] ?? '',
  modelKey: chosenModel.key,
});

Detecting Model Switches

Before forwarding a request to a provider, the routing layer in server/src/services/router.ts invokes maybeInjectContextHandoff. This helper compares the selectedModelKey against the stored lastModelKey for that session. If they differ, the logic determines that a hand-off is required.

Injecting Hand-Off Messages

When a switch is detected, maybeInjectContextHandoff (lines 111–156 in server/src/services/context-handoff.ts) prepends a system message to the conversation array. This message contains:

• A brief description announcing the hand-off
• A summary of the recent conversation (up to approximately 6,000 characters)
• Explicit instructions directing the new model to continue the ongoing task

The modified message array is then forwarded to the new model, allowing it to respond as if it had been part of the conversation from the start.

Configuration and Activation

Context hand-off is opt-in via environment variable. By default, getContextHandoffMode returns 'off', disabling all tracking and injection logic.

# Enable context hand-off on model switches

FREELLMAPI_CONTEXT_HANDOFF=on_model_switch

To check the current mode programmatically:

import { getContextHandoffMode } from './services/context-handoff.js';

const mode = getContextHandoffMode(); // Returns 'off' or 'on_model_switch'

Token Budget Safety and Constraints

Hand-off messages consume tokens, so the implementation enforces strict ceilings to prevent quota violations. The constant MAX_HANDOFF_CHARS (set to 6,000) bounds the summary length, while HANDOFF_MAX_TOKENS (exported at lines 20–24 of server/src/services/context-handoff.ts) provides a pre-computed token ceiling. The routing layer uses this value to reserve tokens during quota checks before invoking the provider.

Practical Implementation Examples

Basic Router Integration

The following pattern shows how the proxy layer integrates the hand-off mechanism during request processing:

import { getContextHandoffMode, maybeInjectContextHandoff } from './services/context-handoff.js';

// Inside the request-proxy logic
const mode = getContextHandoffMode();
const { messages, injected } = maybeInjectContextHandoff({
  mode,
  sessionKey: req.headers['x-session-id'] ?? '',
  messages: req.body.messages,          // Original user/assistant messages
  selectedModelKey: chosenModel.key,    // The model we are about to call
});

if (injected) {
  console.log('Context hand-off injected for model switch');
}
// Forward to provider with potentially modified messages
forwardRequestToProvider({ ...req.body, messages });

Recording Successful Completion

After receiving a successful response from a provider, notify the hand-off service so it can update the last-known model for the session:

import { recordSuccessfulModel } from './services/context-handoff.js';

recordSuccessfulModel({
  sessionKey: req.headers['x-session-id'] ?? '',
  modelKey: chosenModel.key,
});

Testing the Hand-Off Logic

Use the exported test helpers to verify behavior without persisting state between test runs:

import { 
  _clearStoreForTesting, 
  maybeInjectContextHandoff, 
  getContextHandoffMode,
  recordIncomingMessages,
  recordSuccessfulModel 
} from './services/context-handoff.js';

// Reset in-memory store
_clearStoreForTesting();

// Simulate first request handled by modelA
recordIncomingMessages('sess1', [{role:'user', content:'Hello'}]);
recordSuccessfulModel({sessionKey:'sess1', modelKey:'modelA'});

// Simulate switch to modelB
const result = maybeInjectContextHandoff({
  mode: getContextHandoffMode(),
  sessionKey: 'sess1',
  messages: [{role:'user', content:'Continue'}],
  selectedModelKey: 'modelB',
});

console.assert(result.injected === true, 'Hand-off should inject on model switch');

Summary

• Context Preservation: FREELLMAPI_CONTEXT_HANDOFF maintains conversation continuity when FreeLLMAPI routes requests to different models.
• Activation: Set the environment variable to on_model_switch to enable tracking and injection.
• Core Files: Implementation lives in server/src/services/context-handoff.ts, integrated by server/src/services/router.ts.
• Safety: Built-in limits (MAX_HANDOFF_CHARS = 6,000) and token exports prevent quota overruns.
• Storage: In-memory session store keyed by session identifier tracks recent messages and the last successful model.

Frequently Asked Questions

What happens if FREELLMAPI_CONTEXT_HANDOFF is not set?

If the variable is undefined or set to any value other than on_model_switch, getContextHandoffMode returns 'off'. The system skips all tracking and injection logic, and each model request receives only the messages explicitly provided in the API call.

Does context hand-off work across server restarts?

No. The implementation uses an in-memory store (_clearStoreForTesting confirms this is a runtime memory structure). When the Node.js process restarts, the conversation history and last-model mappings are lost. For persistent context, you would need to implement external storage.

How much does the hand-off message increase token usage?

The injected system message is capped at approximately 6,000 characters (MAX_HANDOFF_CHARS), and the service exports HANDOFF_MAX_TOKENS to help the routing layer reserve budget. Typically, the overhead is roughly 1,500 tokens or fewer, depending on conversation density.

Can I customize the hand-off message content?

Currently, the hand-off message template is hardcoded within maybeInjectContextHandoff in server/src/services/context-handoff.ts. To customize the prompt instructions or summary format, you would need to modify the source code in that file, specifically the logic around line 111 where the system message is constructed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}