How OmniRoute's Reasoning Cache Facilitates Replay of Reasoning Content for Multi-Turn Conversations

OmniRoute stores and re-injects model "thinking" so that later turns of a multi-turn conversation can reuse previously generated reasoning, preventing context loss in tool-calling flows.

This capability is essential for providers like DeepSeek that treat reasoning as part of model state. The implementation lives in the Reasoning Cache service at open-sse/services/reasoningCache.ts with durable storage in src/lib/db/reasoningCache.ts.

---

What Gets Cached and Why

When a provider returns a response containing tool-call reasoning (e.g., DeepSeek's "thinking" field), OmniRoute extracts and stores that text keyed by the tool-call ID. Each cache entry contains:

• toolCallId: The unique ID of the tool call generated by the model
• provider: Provider identifier (e.g., deepseek)
• model: Model name (e.g., deepseek-coder-v2)
• reasoning: The raw reasoning string the model produced
• expiresAt: TTL after which the cleanup job evicts the entry

The data lives in two layers: an in-memory Map for fast lookup and SQLite for durability. A periodic job at src/lib/jobs/reasoningCacheCleanupJob.ts removes expired rows.

---

Writing Reasoning to the Cache

The cache is populated inside the chat handling pipeline after receiving an upstream response. In open-sse/services/reasoningCache.ts, the function:

export function cacheReasoning(
  requestId: string,
  provider: string,
  model: string,
  reasoning: string,
) { … }

This is called from the translator/handler when a response includes a reasoning field. The call records reasoning alongside the current request ID, which also serves as a correlation token for the next turn.

---

Detecting When Replay Is Needed

When the next request in the same conversation arrives, OmniRoute checks whether reasoning replay is required:

export function requiresReasoningReplay(params: {
  requestId: string;
  provider: string;
  model: string;
}) { … }

If the provider-model pair requires replay (e.g., DeepSeek models), the service performs a lookup:

export function lookupReasoning(toolCallId: string): string | null { … }

The returned string is injected back into the outbound request payload so the upstream model sees the same "thinking" it generated previously.

---

The Complete Replay Lifecycle

The reasoning cache facilitates replay through a six-step lifecycle:

1. First turn: Model returns a tool-call with reasoning
2. Cache insert: cacheReasoning() stores reasoning keyed by tool-call ID
3. Subsequent turn: requiresReasoningReplay() detects that the provider needs reasoning
4. Lookup: lookupReasoning() retrieves the cached string
5. Injection: Retrieved reasoning is added to the request body before forwarding upstream
6. Cleanup: After the configured TTL, cleanupReasoningCache() removes the entry

This happens transparently to users, who see only a seamless multi-turn tool-calling experience.

---

Why Replay Matters for Multi-Turn Tool Calls

Some providers—notably DeepSeek—treat reasoning as part of model state. When the model issues a tool call, it may later require the same reasoning in a subsequent turn. Without replay, the model loses context, leading to inconsistent or broken tool-calling flows.

The cache guarantees that original reasoning remains available for any later request belonging to the same logical conversation.

---

Implementation Examples

Caching Reasoning After a Tool-Call Response

import { cacheReasoning } from '@/open-sse/services/reasoningCache';

// After receiving a response that contains reasoning:
const requestId = ctx.requestId;       // uniquely generated per request
const provider   = 'deepseek';
const model      = 'deepseek-coder-v2';
const reasoning  = response.choices[0].message.tool_calls[0].reasoning;

await cacheReasoning(requestId, provider, model, reasoning);

Conditionally Injecting Cached Reasoning

import {
  requiresReasoningReplay,
  lookupReasoning,
} from '@/open-sse/services/reasoningCache';

async function maybeInjectReasoning(payload: any, ctx: RequestContext) {
  if (
    await requiresReasoningReplay({
      requestId: ctx.requestId,
      provider: ctx.provider,
      model: ctx.model,
    })
  ) {
    const cached = lookupReasoning(payload.tool_call_id);
    if (cached) {
      // Attach previously stored reasoning so upstream model continues its chain of thought
      payload.reasoning = cached;
    }
  }
  return payload;
}

Running Cache Cleanup

import { cleanupReasoningCache } from '@/open-sse/services/reasoningCache';

// Called by the cron job defined in src/lib/jobs/reasoningCacheCleanupJob.ts
const removed = cleanupReasoningCache(); // returns number of rows deleted
console.log(`Removed ${removed} expired reasoning cache entries`);

---

Key Source Files

File	Purpose
open-sse/services/reasoningCache.ts	Core service: cache insertion, lookup, replay detection, and cleanup
src/lib/db/reasoningCache.ts	SQLite-backed persistence layer
src/lib/jobs/reasoningCacheCleanupJob.ts	Scheduled job for evicting stale entries
tests/unit/reasoning-cache.test.ts	Unit tests verifying caching, lookup, and replay behavior
src/lib/localDb.ts	Exports ReasoningCacheEntry types to the codebase

As implemented in diegosouzapw/OmniRoute, these files constitute the complete Reasoning Replay Cache system.

---

Summary

• Dual-layer storage: In-memory Map for speed, SQLite for durability
• TTL-based eviction: Automatic cleanup via scheduled job at src/lib/jobs/reasoningCacheCleanupJob.ts
• Provider-aware replay: requiresReasoningReplay() determines whether a given provider-model combination needs reasoning injection
• Tool-call ID as key: Enables precise retrieval of reasoning for specific tool invocations
• Transparent operation: Users experience seamless multi-turn conversations without manual intervention

---

Frequently Asked Questions

How does OmniRoute know which conversations need reasoning replay?

OmniRoute calls requiresReasoningReplay() with the request ID, provider, and model. According to the OmniRoute source code, this function checks whether the provider-model pair (e.g., DeepSeek models) is configured to require reasoning replay. If so, the system attempts to retrieve cached reasoning for any tool-call IDs present in the request.

What happens if cached reasoning expires before the next turn?

If lookupReasoning() returns null because the entry was evicted by cleanupReasoningCache(), the request proceeds without injected reasoning. The model may regenerate its reasoning chain, though this can produce slightly different behavior than continuing from the original thinking. The default TTL is configurable to balance storage costs against typical conversation durations.

Is the reasoning cache shared across multiple OmniRoute instances?

No. The cache uses SQLite in src/lib/db/reasoningCache.ts for persistence, but this is typically a local database file per instance. For horizontal scaling, each instance maintains its own cache; reasoning replay only works reliably when subsequent turns route to the same instance. Production deployments may need sticky session configuration or a shared Redis layer for true distribution.

Can I disable reasoning caching entirely?

The caching behavior is integral to correct operation for providers that require replay. However, you can control cleanup frequency by modifying the cron schedule in src/lib/jobs/reasoningCacheCleanupJob.ts and TTL duration through environment configuration. Removing the cache entirely would break multi-turn tool calls for affected providers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}