# How OmniRoute Implements the Reasoning Replay Cache for Multi‑Turn Conversations

> Discover how OmniRoute implements its reasoning replay cache, a hybrid in-memory and SQLite service, to store and re-inject reasoning content for seamless multi-turn conversations with advanced models.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-07-14

---

**OmniRoute’s reasoning replay cache is a hybrid in‑memory and SQLite service that automatically stores a model’s `reasoning_content` from tool‑calling responses and re‑injects it into subsequent requests to satisfy provider contracts for thinking‑mode models like DeepSeek V4 and Kimi‑K2.**

OmniRoute is an open-source AI gateway that routes requests across multiple providers. When handling multi-turn conversations with reasoning models, the API contract typically requires the full `reasoning_content` from previous turns to be passed back in the next request. The **reasoning replay cache** solves this transparently by persisting that content server-side and automatically re‑injecting it, eliminating the need for clients to manually manage large reasoning payloads.

## Architecture of the Reasoning Replay Cache

The implementation lives primarily in [`open-sse/services/reasoningCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/reasoningCache.ts) and follows a layered architecture that balances speed with durability.

### Provider Detection

Before caching occurs, OmniRoute determines if a model requires replay logic. The system checks against a `REASONING_REPLAY_PROVIDERS` set and `REASONING_REPLAY_MODEL_PATTERNS` regex list. Helper functions `isDeepSeekReasoningModel` and `requiresReasoningReplay` (lines 26‑78) evaluate whether the current provider/model combination is subject to the reasoning replay contract.

### Write Path

When an assistant message containing tool calls arrives, `cacheReasoningFromAssistantMessage` extracts the `reasoning_content` string. It delegates to `cacheReasoning` or `cacheReasoningBatch`, which ultimately call `cacheReasoningByKey`. This method writes to both the in‑memory structure and the SQLite persistence layer via `setReasoningCache` (lines 39‑80).

### In‑Memory Cache Layer

The hot path uses a `memoryCache` Map capped at **200 entries**, with a **10 KB limit per entry** and a **2‑hour TTL**. When limits are exceeded, `evictOldest` removes the least recently used entry, while `purgeExpiredMemory` removes stale records (lines 5‑20, 44‑58). This ensures fast lookups for active conversations without unbounded memory growth.

### Read Path

On each new request, `lookupReasoning(toolCallId)` checks the memory cache first. If missed, it falls back to the SQLite table via `getReasoningCache`. Successful DB lookups promote the entry back into memory for future fast hits. The system tracks misses for observability (lines 84‑108).

### Persistence Layer

The SQLite schema is managed in [`src/lib/db/reasoningCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/reasoningCache.ts), which exposes CRUD helpers: `setReasoningCache`, `getReasoningCache`, `deleteReasoningCache`, and `cleanupExpiredReasoning`. A background job in [`src/lib/jobs/reasoningCacheCleanupJob.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/jobs/reasoningCacheCleanupJob.ts) (started from [`src/server-init.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/server-init.ts)) periodically prunes expired rows to prevent database bloat.

## Workflow in Practice

The reasoning replay cache operates transparently across conversation turns:

1. **First turn**: The model returns a tool call with `reasoning_content`. The service extracts this string and stores it under the `tool_call.id` key (or a synthetic `request:<id>:message:<idx>` key when no tool calls are present).
2. **Subsequent turn**: The client sends a new request omitting the previous reasoning. Before forwarding upstream, the router calls `lookupReasoning` for each relevant tool call ID. If found, the cached content is inserted into `requestBody.reasoning_content`.
3. **Replay accounting**: Each successful injection triggers `recordReplay()`, incrementing counters that feed the dashboard metrics.

## Code Implementation Examples

The following patterns demonstrate how to interact with the reasoning replay cache programmatically:

```typescript
// Capture reasoning from an assistant response
import { cacheReasoningFromAssistantMessage } '@/open-sse/services/reasoningCache';

cacheReasoningFromAssistantMessage(
  assistantMessage,           // the raw assistant payload
  providerId,               // e.g., "deepseek"
  modelId,                  // e.g., "deepseek-v4-pro"
  { requestId, messageIndex } // optional context for non‑tool calls
);

```

```typescript
// Re‑inject cached reasoning on the next request
import { lookupReasoning, recordReplay } from '@/open-sse/services/reasoningCache';

const cached = lookupReasoning(toolCallId);
if (cached) {
  // Insert the cached string back into the outgoing request body
  requestBody.reasoning_content = cached;
  recordReplay(); // metrics
}

```

```typescript
// Expose stats to the dashboard
import { getReasoningCacheServiceStats } from '@/open-sse/services/reasoningCache';

const stats = getReasoningCacheServiceStats();
// stats.hits, stats.misses, stats.replayRate, etc.

```

## Configuration and Monitoring

Operators can tune the cache behavior via `REASONING_REPLAY_MODEL_PATTERNS` to add new reasoning models without code changes. The `getReasoningCacheServiceStats` function exposes hit rates, miss counts, and replay volumes, enabling the Settings UI to display real‑time "reasoning replay rate" metrics. The `cleanupExpiredReasoning` database function ensures that entries past their TTL are purged automatically, maintaining query performance as the system scales.

## Summary

- **Hybrid storage**: The reasoning replay cache combines a bounded in‑memory Map (200 entries, 10 KB each, 2‑hour TTL) with durable SQLite storage in [`src/lib/db/reasoningCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/reasoningCache.ts).
- **Automatic detection**: Provider/model combinations are evaluated against `REASONING_REPLAY_PROVIDERS` and regex patterns to determine if replay is required.
- **Transparent injection**: `lookupReasoning` retrieves cached content on the read path and automatically promotes DB hits back into memory for fast subsequent access.
- **Metrics**: The service tracks hits, misses, and replays via `getReasoningCacheServiceStats`, feeding operational dashboards.
- **Maintenance**: A background cleanup job in [`src/lib/jobs/reasoningCacheCleanupJob.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/jobs/reasoningCacheCleanupJob.ts) removes expired entries to prevent storage exhaustion.

## Frequently Asked Questions

### Which models require reasoning replay?

OmniRoute targets **DeepSeek V4**, **Kimi‑K2**, **Xiaomi MiMo**, and other thinking‑mode models that enforce a contract requiring previous `reasoning_content` to be passed back in multi‑turn tool‑calling contexts. The `requiresReasoningReplay` function checks against configurable provider and model patterns to identify these cases dynamically.

### What happens if the cache misses?

If `lookupReasoning` cannot find an entry in memory or the SQLite database, the request proceeds without injection. The upstream provider may return a **400 error** if the model strictly requires the reasoning field, but OmniRoute does not block the request itself. Misses are recorded in the statistics for debugging.

### How long is reasoning content stored?

Entries persist for **2 hours** in the in‑memory cache (subject to LRU eviction) and until explicitly cleaned in the SQLite database. The `cleanupExpiredReasoning` function runs periodically via [`src/lib/jobs/reasoningCacheCleanupJob.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/jobs/reasoningCacheCleanupJob.ts) to remove rows older than the configured TTL, ensuring the database does not grow indefinitely.

### Does the reasoning replay cache impact performance?

The hot path uses constant‑time Map lookups, typically adding less than a millisecond of latency. SQLite operations occur asynchronously on the write path and only on memory misses for reads. The 10 KB entry limit prevents large reasoning blocks from consuming excessive RAM, making the overhead negligible for most production workloads.