How to Implement Reasoning Replay for Multi-Turn Conversations in OmniRoute

OmniRoute implements reasoning replay by caching model reasoning steps in SQLite keyed by tool-call IDs, then injecting cached entries into subsequent conversation turns via the reasoningCache service layer.

OmniRoute is an open-source routing platform that supports 237 AI providers and includes a sophisticated reasoning replay mechanism for multi-turn conversations. This feature stores intermediate reasoning steps in a lightweight SQLite cache, allowing subsequent conversation turns to reference and reuse previous computational results without incurring additional costs or latency.

Understanding the Reasoning Cache Architecture

The implementation follows a three-layer pattern that separates data persistence from business logic and API exposure.

Database Layer in src/lib/db/reasoningCache.ts

The foundation resides in src/lib/db/reasoningCache.ts, which defines the ReasoningCacheEntry schema and provides CRUD helpers including setReasoningCache, getReasoningCache, deleteReasoningCache, clearAllReasoningCache, cleanupExpiredReasoning, getReasoningCacheStats, and getReasoningCacheEntries. Each entry stores the tool-call ID, provider, model, reasoning payload, and expiration timestamp.

Service Layer in open-sse/services/reasoningCache.ts

The business logic lives in open-sse/services/reasoningCache.ts, which orchestrates when to write new entries or retrieve existing ones. This service integrates with the request pipeline to intercept reasoning data after model completion and inject cached reasoning before request translation.

API and Maintenance Components

The HTTP interface in src/app/api/cache/reasoning/route.ts exposes GET, POST, and DELETE endpoints for manual cache management. Background maintenance runs through src/lib/jobs/reasoningCacheCleanupJob.ts, initialized in src/server-init.ts, which periodically evicts expired entries based on a configurable TTL (default 24 hours).

Storing Reasoning Results from the First Turn

When a model completes its initial reasoning step, the handler in open-sse/handlers/chatCore.ts extracts the tool-call ID from request metadata and persists the reasoning object. This captures the full computational context for later replay.

// From open-sse/services/reasoningCache.ts
import { setReasoningCache } from '@/lib/db/reasoningCache';

export async function cacheReasoningIfNeeded(req, resp) {
  const toolCallId = req?.metadata?.toolCallId;
  if (toolCallId && resp?.reasoning) {
    await setReasoningCache({
      toolCallId,
      provider: req.provider,
      model: req.model,
      reasoning: resp.reasoning,
      expiresAt: Date.now() + REASONING_TTL_MS, // Default: 24 hours
    });
  }
}

The REASONING_TTL_MS constant defines the entry lifetime, after which cleanupExpiredReasoning removes stale data during the next cleanup cycle.

Replaying Cached Reasoning in Subsequent Turns

To reuse reasoning in a later turn, the client includes a reasoning field containing the original tool-call ID. The service layer intercepts this request and substitutes the cached payload before the request reaches the upstream provider.

import { getReasoningCache } from '@/lib/db/reasoningCache';

export async function injectCachedReasoning(req) {
  const cachedId = req.body?.reasoning?.toolCallId;
  if (!cachedId) return req;

  const entry = await getReasoningCache(cachedId);
  if (!entry) return req; // Cache miss falls back to fresh computation

  return {
    ...req,
    body: {
      ...req.body,
      reasoning: entry.reasoning, // Deterministic replay
    },
  };
}

Because cache entries are immutable, this replay mechanism produces deterministic results identical to the original computation.

Managing and Monitoring the Cache

HTTP API Endpoints

The REST API in src/app/api/cache/reasoning/route.ts supports manual operations:

# List all cached entries

GET /api/cache/reasoning?token=<api-key>

# Manually insert or update an entry

POST /api/cache/reasoning?token=<api-key>
{
  "toolCallId": "abc123",
  "reasoning": { "effort": "high", "summary": "auto" }
}

# Delete specific entry

DELETE /api/cache/reasoning/<toolCallId>?token=<api-key>

Dashboard Metrics

The React component src/app/(dashboard)/dashboard/cache/components/ReasoningCacheTab.tsx displays operational metrics including reasoningReplayRate (percentage of turns using cached reasoning) and reasoningReplays (total successful cache hits). These statistics derive from getReasoningCacheStats() and provide visibility into cache efficiency across all 237 supported providers.

Complete Multi-Turn Conversation Flow

A typical multi-turn conversation follows this sequence:

1. Turn 1: The client sends an initial request without a tool-call ID. The model computes fresh reasoning, and the service stores the result via setReasoningCache.

2. Turn 2: The client references the previous turn by sending reasoning: {toolCallId: "<id-from-turn-1>"}. The service retrieves the entry via getReasoningCache and injects it into the request, bypassing fresh computation.

3. Turn N: Any subsequent turn can reference the original or a different tool-call ID, enabling multi-turn reasoning replay without additional compute costs or provider latency.

This flow works uniformly across all providers because the cache abstracts provider-specific reasoning formats into a standardized ReasoningCacheEntry structure.

Summary

• OmniRoute stores reasoning steps in SQLite via src/lib/db/reasoningCache.ts using tool-call IDs as primary keys.
• The service layer in open-sse/services/reasoningCache.ts handles automatic caching after model completion and injection before request translation.
• Clients trigger replay by including reasoning.toolCallId in subsequent requests, enabling deterministic multi-turn conversations.
• A configurable TTL (default 24 hours) and background cleanup job prevent unbounded storage growth.
• The dashboard in ReasoningCacheTab.tsx tracks reasoningReplayRate and reasoningReplays to monitor cache efficiency.

Frequently Asked Questions

What is the default TTL for cached reasoning entries?

Cached reasoning entries expire after 24 hours by default. The cleanupExpiredReasoning function in src/lib/db/reasoningCache.ts removes stale entries, triggered by the reasoningCacheCleanupJob.ts scheduler initialized in src/server-init.ts.

How does OmniRoute handle cache misses during replay?

If getReasoningCache returns no entry for a requested tool-call ID, the service layer passes the original request through unchanged. The upstream provider then computes fresh reasoning, and the new result gets cached for future turns if a tool-call ID is present.

Can reasoning be shared across different AI providers?

Yes. The cache schema in src/lib/db/reasoningCache.ts stores the provider and model alongside each entry. While the cache can retrieve entries regardless of provider, the reasoning payload format remains compatible because OmniRoute normalizes reasoning structures across its 237 supported providers during the translation layer.

Where can I view reasoning replay statistics?

The dashboard component ReasoningCacheTab.tsx displays real-time metrics including reasoningReplayRate (percentage of requests using cached reasoning) and reasoningReplays (total hit count). These values calculate from getReasoningCacheStats() and update through the dashboard cache tab interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}