OmniRoute Reasoning Replay System: How It Captures Hidden Model Thinking

OmniRoute's reasoning replay system captures intermediate "thinking" data from providers like DeepSeek V4 and Kimi K2, stores it in an internal SQLite cache, and exposes it via a secure read-only API without leaking reasoning tokens to end-users.

The OmniRoute routing layer includes a specialized mechanism for handling models that output reasoning data separately from final responses. This system allows operators to inspect model thought processes for debugging and analytics while maintaining a clean client-facing API that hides internal deliberation steps.

How the Reasoning Replay System Works

The reasoning replay system operates through four distinct pipeline stages that intercept, store, and mask reasoning content before it reaches the client.

Detection and Injection

When a request targets a reasoning-capable provider, open-sse/translator/helpers/schemaCoercion.ts detects the provider/model combination and automatically injects a hidden reasoning_content field into the request payload. This coercion happens before the request leaves OmniRoute's translation layer, ensuring that providers like DeepSeek V4, Kimi K2, and Moonshot K2 receive the proper schema to emit reasoning streams.

Caching and Storage

As the provider streams back reasoning chunks, open-sse/services/reasoningCache.ts writes these fragments to an in-process SQLite cache keyed by the unique request ID. The cache maintains a persistent record of the model's internal deliberation without blocking the main response stream, allowing the system to accumulate reasoning data asynchronously while continuing to process the primary completion.

Client-Side Hiding

To preserve the standard OpenAI-compatible contract, multiple transformers strip the internal placeholder before it reaches the client. The open-sse/translator/helpers/openaiHelper.ts module replaces the reasoning_content field with an empty object in Chat Completions streams, while open-sse/transformer/responsesTransformer.ts performs similar sanitization for the Responses API. This ensures that existing integrations receive only the final output, not the intermediate thinking steps.

Retrieval API

Developers and operators can access stored reasoning through the paginated endpoint defined in src/app/api/cache/reasoning/route.ts. The /api/cache/reasoning route returns structured entries containing the model name, reasoning tokens, timestamps, and chunk sequences, enabling debugging workflows and analytics dashboards without exposing raw reasoning data to end-users.

Core Implementation Architecture

The reasoning replay system spans six critical files that handle different aspects of the pipeline:

• src/lib/reasoningReplay.ts – Provides the public-facing maybeInjectReasoningReplay() and getReplayForRequest() helpers that orchestrate placeholder injection and cache retrieval
• open-sse/services/reasoningCache.ts – Manages the SQLite-backed storage layer that persists reasoning chunks per request ID
• open-sse/translator/helpers/schemaCoercion.ts – Detects reasoning-capable providers and modifies request schemas to enable replay capture
• open-sse/translator/helpers/openaiHelper.ts – Sanitizes Chat Completions streams by removing the internal replay placeholder
• open-sse/transformer/responsesTransformer.ts – Strips reasoning fields from Responses API output streams
• src/app/api/cache/reasoning/route.ts – Exposes the paginated HTTP interface for querying historical reasoning data

Practical Implementation Examples

Injecting Replay Placeholders

Use the maybeInjectReasoningReplay helper from src/lib/reasoningReplay.ts to conditionally add replay support when translating requests to reasoning-capable models:

import { maybeInjectReasoningReplay } from '@/lib/reasoningReplay';
import { translateRequest } from '@omniroute/open-sse/translator';

const upstreamBody = maybeInjectReasoningReplay({
  model: 'deepseek-v4',
  messages: [{ role: 'user', content: 'Explain quantum entanglement.' }],
});
const transformed = translateRequest(upstreamBody, 'openai', 'deepseek');

Querying Stored Reasoning via API

Retrieve paginated reasoning entries for debugging or admin interfaces by calling the internal cache endpoint:

// GET /api/cache/reasoning?page=1&size=20
fetch('/api/cache/reasoning?page=1&size=20')
  .then(r => r.json())
  .then(data => {
    console.table(data.entries.map(e => ({
      id: e.id,
      model: e.model,
      tokens: e.reasoningTokens,
      created: new Date(e.createdAt).toLocaleString(),
    })));
  });

Direct Cache Access

For internal services that need synchronous access to reasoning data, use the getReplayForRequest function:

import { getReplayForRequest } from '@/lib/reasoningReplay';

const replay = await getReplayForRequest(requestId);
if (replay) {
  console.log('Reasoning chunks:', replay.chunks);
}

Summary

• Detection happens automatically when OmniRoute routes to supported providers like DeepSeek V4, Kimi K2, or Moonshot K2 via schema coercion in schemaCoercion.ts.
• Storage uses SQLite through the reasoningCache.ts service, maintaining an in-process cache keyed by request ID that persists reasoning chunks without affecting latency.
• Client isolation is strict—openaiHelper.ts and responsesTransformer.ts ensure reasoning placeholders never appear in standard Chat Completions or Responses streams.
• Access is read-only via the /api/cache/reasoning endpoint defined in src/app/api/cache/reasoning/route.ts, enabling secure debugging without data leakage.
• Integration is simple using helpers from src/lib/reasoningReplay.ts to inject placeholders and retrieve cached entries programmatically.

Frequently Asked Questions

What providers support reasoning replay in OmniRoute?

The reasoning replay system currently supports providers that offer "reasoning-only" response modes, specifically DeepSeek V4, Kimi K2, and Moonshot K2. The system detects these models in open-sse/translator/helpers/schemaCoercion.ts by checking the provider/model combination against an internal compatibility matrix.

How does OmniRoute prevent reasoning tokens from reaching clients?

OmniRoute employs multiple transformation layers to sanitize output streams. The open-sse/translator/helpers/openaiHelper.ts module replaces the internal reasoning_content placeholder with an empty object for Chat Completions, while open-sse/transformer/responsesTransformer.ts performs equivalent stripping for the Responses API. This ensures that standard client integrations receive only the final completion text, never the intermediate reasoning data.

Can I access reasoning replay data for debugging?

Yes. The system exposes a paginated read-only endpoint at /api/cache/reasoning defined in src/app/api/cache/reasoning/route.ts. This endpoint returns structured entries containing model identifiers, reasoning token counts, timestamps, and chunk sequences. You can also use the getReplayForRequest() helper from src/lib/reasoningReplay.ts for programmatic access within the application codebase.

Is the reasoning replay cache persistent?

The reasoning replay cache uses an in-process SQLite database managed by open-sse/services/reasoningCache.ts. While this provides persistence across requests within a single process lifecycle, the cache is strictly internal to the OmniRoute instance and does not replicate to external systems unless explicitly exported through the cache API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}