How OmniRoute's Session Manager Maintains State Across Retries

OmniRoute maintains per-session state across retries using an in-memory map keyed by deterministic session fingerprints, enabling persistent tool timing, sticky routing, and per-API-key throttling throughout the HTTP request lifecycle.

OmniRoute implements a lightweight session management layer that survives the lifetime of HTTP requests to handle conversation retries gracefully. The system stores session metadata in memory, allowing subsequent retries of the same logical conversation to access previous context without persisting data to disk. This article examines how the sessionManager.ts service implements deterministic session identification, state updates, and lifecycle management to maintain continuity across retries.

Session Fingerprinting and Deterministic ID Generation

OmniRoute generates stable session identifiers using the generateSessionId function defined in open-sse/services/sessionManager.ts (lines 103-146). This function creates a deterministic hash derived from the request payload, including the model name, optional provider, system prompt, first user message, tool signatures, and an optional connectionId header.

Because the hash is deterministic, identical conversation payloads always produce the same session ID. This ensures that when a client retries a request, the system recognizes it as a continuation of the existing session rather than a new conversation.

import { generateSessionId } from '@/open-sse/services/sessionManager';

const body = req.json();               // request payload
const sessionId = generateSessionId(body, {
  provider: 'openai',
  connectionId: req.headers.get('x-connection-id'),
});
// → deterministic 16-byte hex string, reused on retries

Touch Session Pattern for State Persistence

Once generated, the session ID is passed to touchSession (lines 150-164), which implements an upsert pattern. If the session exists, the function updates the lastActive timestamp and increments the requestCount. If the session is new, it initializes a fresh entry in the in-memory map.

Retrieval functions such as getSessionInfo, getSessionConnection, and getActiveSessionCount (lines 90-122) provide read-only access to this metadata for routing logic without exposing the underlying Map structure.

import { touchSession } from '@/open-sse/services/sessionManager';

function handleRequest(req) {
  const sessionId = generateSessionId(req.body, { provider: 'anthropic' });
  touchSession(sessionId, req.headers.get('x-connection-id'));
  // … routing logic …
}

Tracking Tool Execution Across Retries

The session manager preserves tool execution timestamps across request boundaries using markToolFinish and consumeToolFinishTime (lines 168-188). When a tool call completes, markToolFinish stores the current timestamp on the session object. On the subsequent retry, consumeToolFinishTime retrieves and clears this value, enabling accurate calculation of "time-to-first-tool" metrics that span multiple HTTP requests.

import { markToolFinish, consumeToolFinishTime } from '@/open-sse/services/sessionManager';

// After a tool call finishes:
markToolFinish(sessionId);

// On the next request (retry) we can read the elapsed time:
const toolFinishTs = consumeToolFinishTime(sessionId);
if (toolFinishTs) {
  const elapsedMs = Date.now() - toolFinishTs;
  console.log(`TTFT across retry: ${elapsedMs} ms`);
}

Session Lifecycle and Cleanup

Session state persists for a configurable duration via a TTL mechanism implemented in open-sse/services/sessionManager.ts (lines 44-82). A periodic timer executes every minute, evicting sessions whose lastActive timestamp exceeds 15 minutes (configurable via SESSION_TTL_MS). The system also enforces a hard cap of 200 concurrent sessions, automatically removing the oldest entries when this limit is exceeded.

This cleanup strategy prevents memory leaks while ensuring that active retry sequences remain unaffected during brief interruption windows.

Enforcing Per-API-Key Limits

To prevent quota bypasses during retries, OmniRoute tracks session ownership through a secondary Map<string, Set<string>> called activeSessionsByKey. This structure maps API keys to their active session sets, enabling the checkSessionLimit, registerKeySession, and unregisterKeySession functions (lines 41-99) to enforce per-key session quotas.

When a retry arrives, the system checks activeSessionsByKey to determine whether the API key has exceeded its allocated session count, ensuring retries consume existing slots rather than creating unauthorized new sessions.

import {
  checkSessionLimit,
  registerKeySession,
  unregisterKeySession,
} from '@/open-sse/services/sessionManager';

function maybeCreateSession(apiKeyId, sessionId, maxSessions) {
  const limitError = checkSessionLimit(apiKeyId, maxSessions);
  if (limitError) {
    return { error: limitError };
  }
  registerKeySession(apiKeyId, sessionId);
  // …
}

// When the request ends (or TTL expires):
unregisterKeySession(apiKeyId, sessionId);

Summary

• OmniRoute uses deterministic session fingerprinting via generateSessionId to identify conversation retries consistently across multiple HTTP requests.
• The touch session pattern updates lastActive timestamps and request counts, keeping session entries alive during active retry sequences.
• Tool timing persistence through markToolFinish and consumeToolFinishTime enables accurate performance metrics that span request boundaries.
• TTL cleanup runs every minute with a 15-minute expiration window and a 200-session hard cap to prevent memory exhaustion.
• Per-API-key tracking via activeSessionsByKey ensures retries respect quota limits rather than spawning unauthorized new sessions.

Frequently Asked Questions

How does OmniRoute identify the same conversation across retries?

OmniRoute computes a deterministic session ID using generateSessionId in open-sse/services/sessionManager.ts (lines 103-146), which hashes the model name, provider, system prompt, first user message, tool signatures, and optional connection ID. Because identical payloads produce identical hashes, retries automatically map to their existing session state.

What happens to session state when the 15-minute TTL expires?

When a session's lastActive timestamp exceeds the SESSION_TTL_MS threshold (default 15 minutes), the periodic cleanup timer removes the entry from the in-memory map and calls unregisterKeySession to release the API key allocation. Subsequent retries for that conversation generate a new session ID and start fresh state accumulation.

How does the session manager prevent API key quota bypasses during retries?

The system maintains an activeSessionsByKey Map that tracks all sessions associated with each API key. Before creating a new session, checkSessionLimit (lines 41-99) verifies whether the key has reached its maximum allowed sessions. Since retries use the same session ID generated by generateSessionId, they do not trigger new registrations and therefore cannot bypass the per-key limit.

Where is session state stored in OmniRoute?

Session state resides in an in-memory Map structure within the Node.js process running open-sse/services/sessionManager.ts. The system does not persist session data to disk or external databases for standard retry scenarios, relying instead on the TTL-based cleanup mechanism to manage memory usage. For long-lived affinity requirements, the separate src/lib/db/sessionAccountAffinity.ts layer handles database persistence.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}