How OmniRoute's Context-Relay Strategy Hands Off Context Across Targets

OmniRoute's context-relay strategy preserves conversational state across token limits and model changes by detecting exhaustion thresholds, generating structured LLM summaries, and injecting them into subsequent requests as sanitized system messages.

The diegosouzapw/OmniRoute repository implements a sophisticated context-handoff subsystem that solves a critical problem in multi-model AI routing: maintaining continuity when conversations hit constraints. Whether a request approaches a provider's token ceiling or the system switches from gpt-4o to claude-sonnet, OmniRoute ensures no context is lost. This article explains exactly how the context-relay mechanism works, with reference to the actual source implementation.

Detecting When a Context Handoff Is Needed

Every processed request reports its token consumption percentage. The maybeGenerateHandoff function in /open-sse/services/contextHandoff.ts evaluates this against two configurable thresholds:

• HANDOFF_WARNING_THRESHOLD (default 0.85) — triggers proactive handoff generation
• HANDOFF_EXHAUSTION_THRESHOLD (default 0.95) — aborts immediately if already breached

// /open-sse/services/contextHandoff.ts#L11-L13
const HANDOFF_WARNING_THRESHOLD = 0.85;
const HANDOFF_EXHAUSTION_THRESHOLD = 0.95;

// /open-sse/services/contextHandoff.ts#L46-L52
export async function maybeGenerateHandoff(params: HandoffParams): Promise<void> {
  if (params.percentUsed >= HANDOFF_EXHAUSTION_THRESHOLD) {
    throw new HandoffRequiredError("Context exhausted, handoff mandatory");
  }
  if (params.percentUsed < HANDOFF_WARNING_THRESHOLD) {
    return; // No handoff needed
  }
  // Proceed with generation...
}

This early-exit pattern ensures handoffs only fire when genuinely necessary, minimizing unnecessary LLM calls.

Selecting and Condensing Conversation History

Once triggered, selectMessagesForSummary filters the raw message array to fit within MAX_HISTORY_TOKENS_FOR_SUMMARY (8,000 tokens). The algorithm:

1. Preserves all system messages — these carry critical instructions
2. Keeps the most recent non-system messages — prioritizing recency over completeness
3. Trims iteratively until estimateTokens reports compliance

// /open-sse/services/contextHandoff.ts#L33-L45
function selectMessagesForSummary(
  messages: Message[],
  maxTokens: number = MAX_HISTORY_TOKENS_FOR_SUMMARY
): Message[] {
  const systemMsgs = messages.filter(m => m.role === 'system');
  const otherMsgs = messages.filter(m => m.role !== 'system').slice(-20); // Recent window
  
  let selected = [...systemMsgs, ...otherMsgs];
  while (estimateTokens(selected) > maxTokens && otherMsgs.length > 0) {
    otherMsgs.shift(); // Drop oldest
    selected = [...systemMsgs, ...otherMsgs];
  }
  return selected;
}

The token estimator (estimateTokens in contextManager.ts) uses provider-specific tokenizers for accuracy.

Generating Structured Summaries via LLM

The selected messages are formatted and injected into a fixed HANDOFF_PROMPT_TEMPLATE. The LLM receives a low-temperature request that must return valid JSON:

// /open-sse/services/contextHandoff.ts#L25-L39
const HANDOFF_PROMPT_TEMPLATE = `You are a context summarization engine.
Analyze the conversation history and produce a JSON object with these fields:
- summary: string (max 500 chars overview)
- key_decisions: string[] (critical choices made)
- task_progress: string (current completion state)
- active_entities: string[] (people, files, concepts in play)

Conversation:
{formattedMessages}`;

// /open-sse/services/contextHandoff.ts#L82-L88
const completion = await params.handleSingleModel({
  model: params.model,
  messages: [{ role: 'system', content: prompt }],
  temperature: 0.1, // Deterministic output
  response_format: { type: 'json_object' },
}, params.model);

The strict schema ensures downstream components can rely on consistent structure.

Parsing, Sanitizing, and Persisting Handoffs

The parseHandoffJSON function enforces length limits and array normalization:

// /open-sse/services/contextHandoff.ts#L98-L121
function parseHandoffJSON(raw: string): HandoffPayload {
  const parsed = JSON.parse(raw);
  
  // Enforce max lengths to prevent prompt injection
  parsed.summary = parsed.summary?.slice(0, 1000) ?? '';
  parsed.key_decisions = normalizeStringArray(parsed.key_decisions, 10, 200);
  parsed.active_entities = normalizeStringArray(parsed.active_entities, 20, 100);
  
  return {
    summary: parsed.summary,
    key_decisions: parsed.key_decisions,
    task_progress: parsed.task_progress?.slice(0, 500) ?? '',
    active_entities: parsed.active_entities,
    generated_at: Date.now(),
  };
}

The sanitized payload is persisted via upsertHandoff in /src/lib/db/contextHandoffs.ts:

// /src/lib/db/contextHandoffs.ts#L80-L100
export function upsertHandoff(handoff: ContextHandoffRow): void {
  const stmt = db.prepare(`
    INSERT INTO context_handoffs 
      (session_id, combo_name, payload, expires_at, created_at)
    VALUES (?, ?, ?, ?, ?)
    ON CONFLICT(session_id, combo_name) DO UPDATE SET
      payload = excluded.payload,
      expires_at = excluded.expires_at,
      created_at = excluded.created_at
  `);
  stmt.run(
    handoff.session_id,
    handoff.combo_name,
    JSON.stringify(handoff.payload),
    handoff.expires_at ?? Date.now() + (5 * 60 * 60 * 1000), // Default 5h TTL
    Date.now()
  );
}

The 5-hour default TTL prevents stale handoffs from accumulating while allowing reasonable session continuity.

Injecting Handoffs into Subsequent Requests

When a new request begins, injectHandoffIntoBody or injectUniversalHandoffBody retrieves the stored handoff and prepends a specially-crafted system message:

// /open-sse/services/contextHandoff.ts#L78-L95
export function injectHandoffIntoBody(
  body: ChatCompletionRequest,
  handoff: HandoffPayload
): ChatCompletionRequest {
  const handoffMessage: ChatMessage = {
    role: 'system',
    content: `<context_handoff>
Generated: ${new Date(handoff.generated_at).toISOString()}
Summary: ${handoff.summary}
Key Decisions: ${handoff.key_decisions.join(', ')}
Task Progress: ${handoff.task_progress}
Active Entities: ${handoff.active_entities.join(', ')}
</context_handoff>`,
  };
  
  return {
    ...body,
    messages: [handoffMessage, ...body.messages],
  };
}

The XML-tag wrapping (<context_handoff>) clearly demarcates synthetic content from user/assistant turns, helping the receiving model distinguish relayed context from actual conversation history.

Universal Handoff for Model and Provider Switches

Beyond token limits, OmniRoute handles cross-model handoffs when the active provider changes. The maybeGenerateUniversalHandoff function in /open-sse/services/contextHandoff.ts adds provenance tracking:

// /open-sse/services/contextHandoff.ts#L85-L93
export async function maybeGenerateUniversalHandoff(params: UniversalHandoffParams): Promise<void> {
  const existing = getHandoff(params.sessionId, params.comboName);
  
  if (shouldGenerateUniversalHandoff(existing, params.prevModel, params.currModel)) {
    await generateAndStoreUniversalHandoff(params);
  }
}

// /open-sse/services/contextHandoff.ts#L38-L46
function generateAndStoreUniversalHandoff(params: UniversalHandoffParams) {
  const provenance = {
    previous_model: params.prevModel,
    current_model: params.currModel,
    handoff_type: 'universal',
  };
  // Uses same summary pipeline with provenance appended
}

The resulting system message includes model transition metadata, helping the new provider understand not just what happened, but in what context:

// /open-sse/services/contextHandoff.ts#L45-L56
const universalHandoffContent = `<context_handoff type="universal">
Previous Model: ${provenance.previous_model}
Current Model: ${provenance.current_model}
${standardHandoffFields}
</context_handoff>`;

Complete Integration Example

Here's how the pieces connect in a production request cycle:

// 1. End-of-request: check if handoff needed
await maybeGenerateHandoff({
  sessionId: ctx.sessionId,
  comboName: combo.name,
  connectionId: ctx.connectionId,
  percentUsed: usage.percentUsed,  // e.g., 0.87 triggers handoff
  messages: ctx.conversation,
  model: ctx.model,
  expiresAt: null,
  config: combo.contextRelay,
  handleSingleModel: (body, model) => fetchModel(body, model),
});

// 2. Next request: inject stored handoff
const bodyWithHandoff = injectHandoffIntoBody(originalBody, handoffPayload);
const response = await fetchModel(bodyWithHandoff, ctx.model);

// 3. Model switch: universal handoff
await maybeGenerateUniversalHandoff({
  sessionId: ctx.sessionId,
  comboName: combo.name,
  messages: ctx.conversation,
  prevModel: 'gpt-4o',
  currModel: 'claude-3-sonnet-20240229',
  universalConfig: combo.universalHandoff,
  handleSingleModel: (b, m) => fetchModel(b, m),
});

Summary

OmniRoute's context-relay strategy enables seamless conversation continuity through seven coordinated mechanisms:

• Threshold-based detection using configurable exhaustion and warning levels (HANDOFF_WARNING_THRESHOLD, HANDOFF_EXHAUSTION_THRESHOLD)
• Smart history selection that preserves system instructions and recent messages within token budgets
• Structured LLM summarization with strict JSON schemas and low-temperature generation
• Defensive parsing with length limits and array normalization to prevent injection
• SQLite persistence with automatic TTL expiration in the context_handoffs table
• XML-wrapped injection via injectHandoffIntoBody for clear downstream consumption
• Universal handoff support for model/provider migrations with provenance tracking

These components in /open-sse/services/contextHandoff.ts and /src/lib/db/contextHandoffs.ts form a production-ready system for handing off context across arbitrary AI targets without losing conversational coherence.

Frequently Asked Questions

What triggers a context handoff in OmniRoute?

A context handoff triggers when percentUsed (the ratio of consumed to available tokens) crosses HANDOFF_WARNING_THRESHOLD (0.85) or exceeds HANDOFF_EXHAUSTION_THRESHOLD (0.95). The maybeGenerateHandoff function evaluates this after each request completion and aborts immediately if exhaustion is reached.

How does OmniRoute prevent handoff data from growing too large?

The selectMessagesForSummary function caps history at MAX_HISTORY_TOKENS_FOR_SUMMARY (8,000 tokens), and parseHandoffJSON enforces hard limits on every string field (e.g., 1,000 characters for summaries). Additionally, SQLite records expire after a 5-hour TTL by default, preventing unbounded storage growth.

Can the context-relay strategy handle switching between different AI providers?

Yes. The maybeGenerateUniversalHandoff function specifically addresses model and provider switches. It generates a handoff marked with previous_model and current_model provenance, then injects it via injectUniversalHandoffBody. This ensures Claude, GPT-4, or any compatible model receives properly contextualized state regardless of the prior provider.

Where is handoff data stored and how is it retrieved?

Handoff payloads persist in SQLite via /src/lib/db/contextHandoffs.ts, specifically the context_handoffs table keyed by (session_id, combo_name). The upsertHandoff function handles both inserts and updates, while retrieval uses getHandoff(sessionId, comboName) to fetch the most recent payload for injection into new requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}