How OmniRoute's Reasoning Replay Feature Works and Caches Reasoning Traces

OmniRoute's reasoning replay captures a model's internal chain-of-thought during multi-turn tool-calling flows and stores it in a hybrid in-memory/SQLite cache for deterministic debugging, audit trails, and optional encrypted replay.

The reasoning replay feature in the diegosouzapw/OmniRoute codebase solves a critical problem in production LLM systems: reproducing exactly what a model was "thinking" during complex tool-calling sessions. When enabled, every reasoning step gets persisted with configurable TTL and can be injected back into response streams for downstream consumers.

What Reasoning Replay Actually Records

Unlike standard logging, reasoning replay captures the raw reasoning text that models generate when deciding which tools to call. This includes the logical steps between receiving a user query and emitting a tool_calls or assistant message.

The feature operates at three architectural layers according to the source code:

Layer	Responsibility	Key File
Capture	Translator inserts placeholders; stores reasoning when provider returns	open-sse/services/reasoningCache.ts
Persistence	Hybrid in-memory + SQLite storage with durable schema	src/lib/db/reasoningCache.ts
Cleanup	Background job evicts expired entries from both stores	src/lib/jobs/reasoningCacheCleanupJob.ts

How Reasoning Traces Are Cached

The caching mechanism follows a four-step flow implemented across the service layer.

1. Scope Resolution

Each trace is keyed by a reasoning cache scope combining requestId and optional interleaved capability flags. The requiresReasoningReplay helper determines if a request qualifies for capture.

2. Dual-Write to Cache

The cacheReasoning() method in open-sse/services/reasoningCache.ts writes simultaneously to:

• In-memory Map: Fast lookup for active sessions
• SQLite table: Durable storage with fields requestId, modelId, scope, reasoning, createdAt, expiresAt

3. Cache Retrieval for Replay

When a subsequent turn requests previous reasoning, getReasoningCache(scope) returns the stored text. Downstream components hydrate this into the response stream as a placeholder, maintaining message ordering.

4. TTL-Based Expiration

Default TTL is approximately 30 minutes. The cleanup job prunes both stores, preventing unbounded memory growth in long-running deployments.

Enabling and Using Reasoning Replay

API-Level Activation

Pass reasoningReplay: true in requests to /api/v1/responses (handled in src/app/api/v1/responses/route.ts):

// Enable reasoning replay for a multi-turn session
const response = await fetch("https://api.omniroute.dev/api/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "gpt-4o",
    messages: conversationHistory,
    reasoningReplay: true  // Opt-in flag
  })
});

Programmatic Cache Interaction

import { reasoningCache, ReasoningCacheScope } from "@/open-sse/services/reasoningCache";

// Define scope for this conversation
const scope: ReasoningCacheScope = {
  requestId: "chat-session-abc123",
  model: "gpt-4o",
};

// After receiving reasoning from provider
await reasoningCache.cacheReasoning(
  scope,
  "openai",           // provider
  "gpt-4o",           // model
  "Analyzing user intent →确定为 profile lookup → selecting getUserProfile tool"
);

// Later turn: retrieve and replay
const cachedReasoning = await reasoningCache.getReasoningCache(scope);
if (cachedReasoning) {
  stream.write({ role: "assistant", reasoning: cachedReasoning });
}

Administrative Operations

// Clear all cached reasoning (testing/emergency)
import { clearReasoningCacheAll } from "@/open-sse/services/reasoningCache";
await clearReasoningCacheAll();  // Wipes memory + SQLite

// Check cache health
const health = await fetch("/api/cache/reasoning").then(r => r.json());
// { entries: 42, sizeBytes: 123456, expiresSoon: 3 }

Storage Implementation Details

The SQLite schema in src/lib/db/reasoningCache.ts implements the durable layer:

• Indexed on (requestId, scope) for fast scoped queries
• expiresAt column enables efficient range-based deletion
• Cross-process safe for horizontally-scaled deployments

The in-memory layer uses a Map with identical TTL semantics, providing sub-millisecond reads for hot entries while offloading cold data to SQLite.

Configuration and Capabilities

Setting	Control	Default
Global toggle	INTERLEAVED_REASONING_REPLAY capability	Disabled
Per-request	reasoningReplay body parameter	false
Encryption	"encrypted reasoning replay" capability (PR #9876)	Disabled
TTL	Environment override	~30 minutes

Summary

• Reasoning replay captures model chain-of-thought during tool-calling flows for deterministic reconstruction
• Hybrid caching combines in-memory speed with SQLite durability in open-sse/services/reasoningCache.ts and src/lib/db/reasoningCache.ts
• Automatic cleanup via reasoningCacheCleanupJob.ts prevents resource leaks with configurable TTL
• Opt-in activation through API flag or global capability, with encrypted variant available

Frequently Asked Questions

What is the reasoning replay feature in OmniRoute?

Reasoning replay is a debugging and audit feature that records the internal reasoning text models generate during multi-turn tool-calling conversations. It enables developers to replay exactly what a model was thinking at any point in a session, useful for troubleshooting complex agent behaviors and maintaining compliance records.

How does OmniRoute cache reasoning traces?

OmniRoute uses a hybrid in-memory + SQLite architecture. The reasoningCache service in open-sse/services/reasoningCache.ts writes every trace to both a fast in-memory Map and a durable SQLite table managed by src/lib/db/reasoningCache.ts. Entries include metadata like requestId, modelId, scope, and expiration timestamp for precise retrieval.

Can reasoning replay be encrypted?

Yes. An encrypted reasoning replay variant exists, gated by a dedicated capability flag introduced in PR #9876. When enabled, reasoning text is encrypted at rest in SQLite and only decrypted when retrieved by authorized scopes, protecting sensitive model reasoning in production environments.

How do I clear the reasoning cache?

Use the clearReasoningCacheAll() function exported from open-sse/services/reasoningCache.ts to wipe both in-memory and SQLite stores. This is useful in test suites, development environments, or emergency situations requiring immediate cache invalidation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}