How OmniRoute Implements the Reasoning Replay Cache for Multi‑Turn Conversations

OmniRoute’s reasoning replay cache is a hybrid in‑memory and SQLite service that automatically stores a model’s reasoning_content from tool‑calling responses and re‑injects it into subsequent requests to satisfy provider contracts for thinking‑mode models like DeepSeek V4 and Kimi‑K2.

OmniRoute is an open-source AI gateway that routes requests across multiple providers. When handling multi-turn conversations with reasoning models, the API contract typically requires the full reasoning_content from previous turns to be passed back in the next request. The reasoning replay cache solves this transparently by persisting that content server-side and automatically re‑injecting it, eliminating the need for clients to manually manage large reasoning payloads.

Architecture of the Reasoning Replay Cache

The implementation lives primarily in open-sse/services/reasoningCache.ts and follows a layered architecture that balances speed with durability.

Provider Detection

Before caching occurs, OmniRoute determines if a model requires replay logic. The system checks against a REASONING_REPLAY_PROVIDERS set and REASONING_REPLAY_MODEL_PATTERNS regex list. Helper functions isDeepSeekReasoningModel and requiresReasoningReplay (lines 26‑78) evaluate whether the current provider/model combination is subject to the reasoning replay contract.

Write Path

When an assistant message containing tool calls arrives, cacheReasoningFromAssistantMessage extracts the reasoning_content string. It delegates to cacheReasoning or cacheReasoningBatch, which ultimately call cacheReasoningByKey. This method writes to both the in‑memory structure and the SQLite persistence layer via setReasoningCache (lines 39‑80).

In‑Memory Cache Layer

The hot path uses a memoryCache Map capped at 200 entries, with a 10 KB limit per entry and a 2‑hour TTL. When limits are exceeded, evictOldest removes the least recently used entry, while purgeExpiredMemory removes stale records (lines 5‑20, 44‑58). This ensures fast lookups for active conversations without unbounded memory growth.

Read Path

On each new request, lookupReasoning(toolCallId) checks the memory cache first. If missed, it falls back to the SQLite table via getReasoningCache. Successful DB lookups promote the entry back into memory for future fast hits. The system tracks misses for observability (lines 84‑108).

Persistence Layer

The SQLite schema is managed in src/lib/db/reasoningCache.ts, which exposes CRUD helpers: setReasoningCache, getReasoningCache, deleteReasoningCache, and cleanupExpiredReasoning. A background job in src/lib/jobs/reasoningCacheCleanupJob.ts (started from src/server-init.ts) periodically prunes expired rows to prevent database bloat.

Workflow in Practice

The reasoning replay cache operates transparently across conversation turns:

1. First turn: The model returns a tool call with reasoning_content. The service extracts this string and stores it under the tool_call.id key (or a synthetic request:<id>:message:<idx> key when no tool calls are present).
2. Subsequent turn: The client sends a new request omitting the previous reasoning. Before forwarding upstream, the router calls lookupReasoning for each relevant tool call ID. If found, the cached content is inserted into requestBody.reasoning_content.
3. Replay accounting: Each successful injection triggers recordReplay(), incrementing counters that feed the dashboard metrics.

Code Implementation Examples

The following patterns demonstrate how to interact with the reasoning replay cache programmatically:

// Capture reasoning from an assistant response
import { cacheReasoningFromAssistantMessage } '@/open-sse/services/reasoningCache';

cacheReasoningFromAssistantMessage(
  assistantMessage,           // the raw assistant payload
  providerId,               // e.g., "deepseek"
  modelId,                  // e.g., "deepseek-v4-pro"
  { requestId, messageIndex } // optional context for non‑tool calls
);

// Re‑inject cached reasoning on the next request
import { lookupReasoning, recordReplay } from '@/open-sse/services/reasoningCache';

const cached = lookupReasoning(toolCallId);
if (cached) {
  // Insert the cached string back into the outgoing request body
  requestBody.reasoning_content = cached;
  recordReplay(); // metrics
}

// Expose stats to the dashboard
import { getReasoningCacheServiceStats } from '@/open-sse/services/reasoningCache';

const stats = getReasoningCacheServiceStats();
// stats.hits, stats.misses, stats.replayRate, etc.

Configuration and Monitoring

Operators can tune the cache behavior via REASONING_REPLAY_MODEL_PATTERNS to add new reasoning models without code changes. The getReasoningCacheServiceStats function exposes hit rates, miss counts, and replay volumes, enabling the Settings UI to display real‑time "reasoning replay rate" metrics. The cleanupExpiredReasoning database function ensures that entries past their TTL are purged automatically, maintaining query performance as the system scales.

Summary

• Hybrid storage: The reasoning replay cache combines a bounded in‑memory Map (200 entries, 10 KB each, 2‑hour TTL) with durable SQLite storage in src/lib/db/reasoningCache.ts.
• Automatic detection: Provider/model combinations are evaluated against REASONING_REPLAY_PROVIDERS and regex patterns to determine if replay is required.
• Transparent injection: lookupReasoning retrieves cached content on the read path and automatically promotes DB hits back into memory for fast subsequent access.
• Metrics: The service tracks hits, misses, and replays via getReasoningCacheServiceStats, feeding operational dashboards.
• Maintenance: A background cleanup job in src/lib/jobs/reasoningCacheCleanupJob.ts removes expired entries to prevent storage exhaustion.

Frequently Asked Questions

Which models require reasoning replay?

OmniRoute targets DeepSeek V4, Kimi‑K2, Xiaomi MiMo, and other thinking‑mode models that enforce a contract requiring previous reasoning_content to be passed back in multi‑turn tool‑calling contexts. The requiresReasoningReplay function checks against configurable provider and model patterns to identify these cases dynamically.

What happens if the cache misses?

If lookupReasoning cannot find an entry in memory or the SQLite database, the request proceeds without injection. The upstream provider may return a 400 error if the model strictly requires the reasoning field, but OmniRoute does not block the request itself. Misses are recorded in the statistics for debugging.

How long is reasoning content stored?

Entries persist for 2 hours in the in‑memory cache (subject to LRU eviction) and until explicitly cleaned in the SQLite database. The cleanupExpiredReasoning function runs periodically via src/lib/jobs/reasoningCacheCleanupJob.ts to remove rows older than the configured TTL, ensuring the database does not grow indefinitely.

Does the reasoning replay cache impact performance?

The hot path uses constant‑time Map lookups, typically adding less than a millisecond of latency. SQLite operations occur asynchronously on the write path and only on memory misses for reads. The 10 KB entry limit prevents large reasoning blocks from consuming excessive RAM, making the overhead negligible for most production workloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}