OmniRoute Reasoning Replay Feature: How It Works and Why It Matters

OmniRoute's Reasoning Replay feature automatically caches and resends reasoning_content from "thinking-mode" LLM providers like DeepSeek V4 and Kimi K2, preventing 400 errors without requiring client-side code changes.

The reasoning replay feature in OmniRoute solves a critical API contract problem with modern reasoning-enabled language models. Certain providers require downstream clients to echo back the reasoning_content field from previous assistant responses on every subsequent turn. OmniRoute makes this requirement transparent through an intelligent caching and injection system implemented across its request pipeline.

---

Why Reasoning Replay Is Necessary

Several "thinking-mode" providers—including DeepSeek V4, Kimi K2, and Xiaomi MiMo—enforce strict conversational state rules. If a client sends a request containing a previous assistant message that originally included reasoning_content, but omits that field in the current turn, the upstream API returns a 400 error: "The reasoning_content in the thinking mode must be passed back to the API."

This creates a burden for applications integrating these models. Without automation, developers must manually extract, store, and re-inject reasoning fields across multi-turn conversations. OmniRoute's reasoning replay system eliminates this complexity entirely.

---

How Reasoning Replay Works: Four Core Layers

The feature operates through four loosely-coupled layers inside the OmniRoute request pipeline.

1. Provider and Model Detection

Before any request modification occurs, OmniRoute determines whether the target provider requires reasoning replay.

The requiresReasoningReplay function in open-sse/services/reasoningCache.ts (lines 26-88) applies three detection strategies:

• A whitelist of provider names (REASONING_REPLAY_PROVIDERS)
• Regex patterns matching model identifiers (REASONING_REPLAY_MODEL_PATTERNS)
• An explicit interleavedField === "reasoning_content" configuration flag

This multi-layer detection ensures new reasoning-capable models are recognized without code changes when their patterns are added to the configuration.

2. Request-Side Placeholder Injection

When a provider requires replay and the outgoing messages contain an assistant-side tool call lacking reasoning_content, OmniRoute injects an empty string placeholder. This satisfies the provider's JSON schema validation while remaining invisible to downstream clients.

The injectEmptyReasoningContentForToolCalls function in open-sse/translator/helpers/schemaCoercion.ts handles this:

• Lines 70-84: Validation and conditional logic
• Lines 89-100: The actual mutation adding reasoning_content: ""

The placeholder ensures the upstream provider receives a valid request shape even on the first interaction when no cached reasoning exists yet.

3. Caching and Replay Mechanism

When a model returns reasoning_content, OmniRoute extracts and stores it for future retrieval. The cache uses a hybrid in-memory + SQLite architecture optimized for low-latency lookups across request boundaries.

Cache write path: The cacheReasoningFromAssistantMessage function (lines 245-283) walks assistant messages, identifies reasoning fields, and stores them keyed by tool-call ID or synthetic message identifiers.

Cache read path: The lookupReasoning function (lines 292-330) implements a three-tier lookup:

1. Fast in-memory map check
2. SQLite fall-back via getReasoningCache
3. Promotion of DB hits back to memory for subsequent requests

On miss, the function returns null and the placeholder injection layer handles the fallback.

The recordReplay utility tracks replay events for observability, feeding metrics to the dashboard API.

4. Observability and Management

Operators monitor reasoning replay health through a dedicated REST endpoint at /api/cache/reasoning. This route (lines 15-41) exposes:

• Cache entry counts (memory vs. database)
• Hit/miss counters
• Replay rate statistics
• Paginated entry inspection

Periodic auto-cleanup (lines 472-520) prevents unbounded cache growth by evicting stale entries on a configurable schedule.

---

Reasoning Replay in Practice: Code Examples

Example 1: First Request to a DeepSeek V4 Model

import { createChatCompletion } from '@omniroute/open-sse/client';

await createChatCompletion({
  provider: 'deepseek',
  model: 'deepseek-v4-flash',
  messages: [
    {
      role: 'assistant',
      content: 'Here is my answer.',
      tool_calls: [{
        id: 'call_123',
        type: 'function',
        function: { name: 'calc', arguments: '{}' }
      }],
    },
  ],
});

The client deliberately omits reasoning_content. OmniRoute's translator detects that deepseek-v4-flash requires replay and automatically injects reasoning_content: "" before sending to the upstream provider.

Example 2: Second Turn with Automatic Replay

await createChatCompletion({
  provider: 'deepseek',
  model: 'deepseek-v4-flash',
  messages: [
    {
      role: 'assistant',
      content: 'Continuing the task.',
      tool_calls: [{
        id: 'call_123',  // Same tool call ID as before
        type: 'function',
        function: { name: 'calc', arguments: '{}' }
      }],
    },
  ],
});

If the first response contained reasoning_content, cacheReasoningFromAssistantMessage stored it under key call_123. On this second request, lookupReasoning('call_123') retrieves the cached string and OmniRoute inserts it into the request body. The provider contract is satisfied without any client-side boilerplate.

Example 3: Querying Cache Statistics

import { fetch } from 'node-fetch';

const res = await fetch('http://localhost:20128/api/cache/reasoning', {
  headers: { Authorization: 'Bearer <admin-token>' },
});

const { stats, entries } = await res.json();
console.log('Reasoning cache hit-rate:', stats.replayRate);
console.log('Memory entries:', stats.memoryEntries);
console.log('DB entries:', stats.dbEntries);

The endpoint uses getReasoningCacheServiceStats to aggregate metrics across the hybrid cache layers.

---

Complete Request Flow

client → OmniRoute request
       → (1) Provider detection (requiresReasoningReplay)
       → (2) Empty reasoning_content placeholder injection (if needed)
       → upstream LLM
       ← response (may contain reasoning_content)
       → (3) Cache extraction (cacheReasoningFromAssistantMessage)
       ← next client request
       → Cache lookup (lookupReasoning)
       → Replay insertion
       → upstream LLM

Because placeholder injection and cache operations run inside OmniRoute's translator and executor layers, downstream consumers—including the Responses API, chat UIs, and MCP tool integrations—never encounter the internal reasoning_content field. The placeholder is stripped before final response delivery, keeping reasoning replay an implementation detail.

---

Summary

• Reasoning replay in OmniRoute automates the reasoning_content echo requirement enforced by DeepSeek V4, Kimi K2, Xiaomi MiMo, and similar "thinking-mode" providers.

• Three technical layers implement the feature: provider detection (requiresReasoningReplay), placeholder injection (injectEmptyReasoningContentForToolCalls), and hybrid caching (cacheReasoningFromAssistantMessage / lookupReasoning).

• Zero client impact: Applications using OmniRoute need no code changes to support reasoning models; the system handles schema compliance transparently.

• Observability built‑in: The /api/cache/reasoning endpoint and automatic metrics collection enable production monitoring of cache health and replay rates.

• Key files: open-sse/services/reasoningCache.ts (detection, cache core), open-sse/translator/helpers/schemaCoercion.ts (placeholder injection), and src/app/api/cache/reasoning/route.ts (management API).

---

Frequently Asked Questions

What happens if the reasoning cache lookup misses?

When lookupReasoning returns null for a tool-call ID, the request pipeline falls back to the placeholder injection layer. An empty string reasoning_content: "" is added to the assistant message, satisfying the provider's schema validation. The conversation continues normally, and any reasoning content returned in the response is cached for subsequent turns.

Does reasoning replay work with streaming responses?

Yes. The cache extraction logic in cacheReasoningFromAssistantMessage processes complete assistant messages regardless of whether they arrived via streaming or batch responses. The caching layer operates on the finalized message structure before it reaches the client, ensuring reasoning content is captured and available for replay on the next turn.

How is cache persistence handled across server restarts?

The hybrid cache design uses SQLite for durable storage. In-memory entries are lost on restart, but lookupReasoning automatically falls back to the database. When a DB hit occurs, the entry is promoted back to the in-memory map, restoring fast-path performance without manual intervention.

Can operators disable reasoning replay for specific providers?

While the whitelist and regex patterns in open-sse/services/reasoningCache.ts control detection, operators can also influence behavior through provider configuration. Setting interleavedField to a value other than "reasoning_content" for a specific provider causes requiresReasoningReplay to return false, bypassing the entire mechanism for that provider's requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}