How OmniRoute's Reasoning Replay Re‑Injects Cached Reasoning for Multi‑Turn Conversations

OmniRoute's Reasoning Replay Cache automatically stores a model's reasoning_content when tool‑calling responses are received and re‑injects that content on later turns where the client omits it, satisfying the strict "reasoning must be passed back" contract required by DeepSeek V4, Kimi‑K2, and other thinking‑mode models.

OmniRoute's reasoning replay mechanism solves a critical compatibility problem in multi‑turn conversations with modern LLMs that expose their internal reasoning chains. When models like DeepSeek V4 return reasoning_content alongside tool calls, subsequent API calls must include that reasoning or face 400 errors. OmniRoute's hybrid in‑memory and SQLite cache makes this requirement transparent to clients by automatically preserving and replaying reasoning across conversation turns.

How Reasoning Replay Detection Works

OmniRoute identifies which provider and model combinations require reasoning replay through two parallel mechanisms in open-sse/services/reasoningCache.ts.

The REASONING_REPLAY_PROVIDERS set and REASONING_REPLAY_MODEL_PATTERNS regex list define eligible targets, with helper functions isDeepSeekReasoningModel and requiresReasoningReplay performing the actual classification (lines 26-78).

This detection runs on every request to determine whether reasoning storage and replay logic should activate.

The Write Path: Capturing Reasoning Content

When a tool‑calling response arrives, cacheReasoningFromAssistantMessage extracts the reasoning_content string and persists it through cacheReasoning or cacheReasoningBatch. These delegate to cacheReasoningByKey, which writes to both storage layers (lines 39-80).

// Capture reasoning from an assistant response
import { cacheReasoningFromAssistantMessage } from '@/open-sse/services/reasoningCache';

cacheReasoningFromAssistantMessage(
  assistantMessage,           // the raw assistant payload from upstream
  providerId,                // e.g. "deepseek"
  modelId,                   // e.g. "deepseek-v4-pro"
  { requestId, messageIndex } // context for synthetic keys when no tool calls present
);

Keys are generated from tool_call.id when available, or constructed as request:<id>:message:<idx> for non‑tool responses. This dual‑key strategy ensures reasoning survives regardless of response structure.

In‑Memory Cache Configuration

The hot path uses a memory‑first design with strict resource boundaries to prevent unbounded growth:

Parameter	Value	Purpose
Maximum entries	200	Prevents memory exhaustion under load
Per‑entry limit	10 KB	Caps individual reasoning payload size
TTL	2 hours	Automatic expiration of stale entries

The memoryCache Map, evictOldest eviction policy, and purgeExpiredMemory cleanup run continuously on access patterns (lines 5-20, 44-58).

The Read Path: Re‑Injecting Reasoning on Subsequent Turns

Before forwarding client requests upstream, OmniRoute calls lookupReasoning(toolCallId) to check for cached reasoning from previous turns. The lookup follows a memory‑first, DB‑fallback pattern with automatic promotion on hit (lines 84-108).

// Re‑inject cached reasoning on the next request
import { lookupReasoning, recordReplay } from '@/open-sse/services/reasoningCache';

const cached = lookupReasoning(toolCallId);
if (cached) {
  // Satisfy the provider's reasoning_content requirement
  requestBody.reasoning_content = cached;
  recordReplay(); // increment replay counter for metrics
}

If found, the cached string is inserted into the outgoing request payload and the entry is promoted back to memory for future fast access. Misses are explicitly recorded for observability.

SQLite Persistence for Cross‑Process Reliability

The database layer in src/lib/db/reasoningCache.ts provides durability across server restarts and horizontal scaling scenarios. Core operations include:

• setReasoningCache – write or update cached reasoning
• getReasoningCache – retrieve by key with optional expiration filtering
• deleteReasoningCache – manual invalidation
• cleanupExpiredReasoning – scheduled pruning of stale rows

The cleanup job at src/lib/jobs/reasoningCacheCleanupJob.ts runs periodically via src/server-init.ts to maintain table health.

Multi‑Turn Conversation Workflow

A complete reasoning replay cycle proceeds through three phases:

1. First turn – Model returns tool calls with reasoning_content. OmniRoute extracts and caches this under each tool_call.id.

2. Subsequent turn – Client sends follow‑up without reasoning_content. The router intercepts the request, looks up cached reasoning by tool call ID, and injects it before upstream delivery.

3. Replay accounting – Each injection increments the replays counter, enabling dashboard visibility into replay frequency and cache effectiveness.

This automation eliminates the need for clients to manually track and resubmit reasoning chains across conversation turns.

Metrics and Observability

The service exposes operational telemetry through getReasoningCacheServiceStats and getReasoningCacheServiceEntries (lines 39-54, 44-52).

// Expose stats to the dashboard
import { getReasoningCacheServiceStats } from '@/open-sse/services/reasoningCache';

const stats = getReasoningCacheServiceStats();
// stats.hits, stats.misses, stats.replayRate, etc.

These power the Settings UI and administrative dashboards, showing cache hit rates and reasoning replay frequency per model.

Key Implementation Files

File	Responsibility
open-sse/services/reasoningCache.ts	Core hybrid cache with memory + DB coordination
src/lib/db/reasoningCache.ts	SQLite schema and CRUD operations
src/lib/jobs/reasoningCacheCleanupJob.ts	Scheduled expiration of database rows
src/server-init.ts	Job initialization on server startup
tests/unit/service-reasoning-cache.test.ts	Unit tests for cache behavior
tests/unit/reasoning-replay-big-pickle.test.ts	Integration tests for complex multi‑turn scenarios

Summary

• Hybrid storage combines in‑memory speed (200 entries, 2‑hour TTL) with SQLite durability for reliable reasoning replay across server restarts.

• Automatic detection via requiresReasoningReplay eliminates manual configuration for compatible providers and models.

• Transparent injection on the read path satisfies upstream API contracts without client code changes.

• Full observability through hit/miss/replay metrics enables operational monitoring of cache performance.

Frequently Asked Questions

What happens if reasoning content exceeds the 10 KB per‑entry limit?

Entries larger than 10 KB are rejected from the in‑memory cache but may still persist to SQLite depending on database configuration. The service prioritizes availability over strict size enforcement for critical reasoning chains.

Does reasoning replay work for non‑tool‑calling responses?

Yes. When no tool_call.id exists, OmniRoute generates synthetic keys using request:<id>:message:<idx> format, enabling reasoning replay for standard assistant messages that expose thinking content.

How does OmniRoute handle cache expiration mid‑conversation?

Expired entries are purged from memory on access and cleaned from SQLite by the periodic job. If a lookup occurs after expiration, the cache miss triggers normal request forwarding without reasoning injection—clients may receive a provider error in strict‑mode configurations.

Can multiple OmniRoute instances share reasoning cache state?

Yes. The SQLite persistence layer enables cross‑process sharing when instances connect to the same database. In‑memory caches remain instance‑local, with the database serving as the consistent source of truth for reasoning replay across deployments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}