How OmniRoute's Reasoning Cache Works for Multi-Turn Conversations with Strict Providers

OmniRoute implements a three-layer Reasoning Replay Cache—combining an in-memory LRU store, SQLite persistence, and a background cleanup job—to automatically capture and re-inject reasoning_content blocks, enabling seamless multi-turn conversations with strict thinking-mode providers like DeepSeek and Kimi.

OmniRoute acts as a unified gateway for diverse LLM providers, but "thinking-mode" models such as DeepSeek V4, Kimi K2, and Qwen-Thinking enforce strict requirements: they return a reasoning_content block that must be echoed back in subsequent requests or the upstream returns a 400 error. The reasoning cache eliminates this friction by storing assistant reasoning persistently and replaying it automatically without client-side management.

The Three-Layer Cache Architecture

OmniRoute’s reasoning cache operates through a tiered storage system designed for speed, durability, and automatic maintenance.

In-Memory LRU Store

The fastest layer is a capped in-memory map living in open-sse/services/reasoningCache.ts. This LRU-style cache holds up to 2,000 entries and provides immediate lookup for the current process. It serves as the first check during request translation, minimizing latency for active conversations.

SQLite Persistence

For durability across restarts, the system writes every entry to src/lib/db/reasoningCache.ts. This module manages a SQLite table storing tool_call_id → reasoning_content mappings along with metadata including provider, model, and an expires_at timestamp (default 24 hours). This layer ensures that even if the OmniRoute process restarts, multi-turn conversations can continue without losing reasoning context.

Background Cleanup Job

To prevent unbounded growth, src/lib/jobs/reasoningCacheCleanupJob.ts runs a periodic sweep. It respects the OMNIROUTE_REASONING_CACHE_CLEANUP_INTERVAL_MS environment variable (defaulting to 30 minutes) to purge expired rows from both the database and in-memory structures.

Capturing Reasoning Content

When a strict provider returns a response, OmniRoute extracts and persists the reasoning content before returning the result to the client.

In open-sse/handlers/chatCore.ts, the handler cacheReasoningFromAssistantMessage processes each assistant message containing tool_calls. It extracts the reasoning_content field (falling back to the legacy reasoning field for compatibility) and persists it via setReasoningCache. This function atomically updates both the in-memory map and the SQLite table, recording the raw reasoning text alongside the toolCallId, provider, model, and expiration timestamp.

Replaying Content for Strict Providers

During request translation, OmniRoute determines whether the target provider requires reasoning replay and hydrates the outgoing payload accordingly.

The helper requiresReasoningReplay in open-sse/services/reasoningCache.ts checks hard-coded allow-lists (the REASONING_REPLAY_PROVIDERS set) to identify strict models. If matched, the system iterates through prior assistant messages and calls lookupReasoning for each tool_call_id:

if (requiresReasoningReplay({ provider, model })) {
    const cached = lookupReasoning(params);
    if (cached) {
        msg.reasoning_content = cached;
        recordReplay(); // Bump hit statistics
    } else {
        msg.reasoning_content = ""; // Prevent 400 errors on miss
    }
}

• Cache hits inject the exact stored reasoning block.
• Cache misses populate the field with an empty string to maintain API contract validity, avoiding the "Param Incorrect" error.
• Non-strict providers bypass the cache entirely, with reasoning_content stripped to avoid quadratic token growth.

Configuration and Observability

OmniRoute exposes management endpoints and environment variables to monitor and control the cache behavior.

Environment Variables

• OMNIROUTE_REASONING_CACHE_CLEANUP_INTERVAL_MS: Controls the cleanup job frequency (default: 1800000ms/30min).
• Default TTL: 24 hours for entries before automatic expiration.

Management API Endpoints

Inspect cache status:

curl "https://your-omniroute.local/api/cache/reasoning"

Returns JSON with hit/miss counters, entry counts, and recent entries:

{
  "hits": 124,
  "misses": 7,
  "entries": [
    {
      "toolCallId": "call_abc123",
      "provider": "deepseek",
      "model": "deepseek-v4",
      "reasoning": "... captured reasoning ..."
    }
  ]
}

Clear cache by provider:

curl -X DELETE "https://your-omniroute.local/api/cache/reasoning?provider=deepseek"

Implementation Examples

Adding a New Strict Provider

To enable reasoning replay for additional providers, extend the allow-list in open-sse/services/reasoningCache.ts:

const REASONING_REPLAY_PROVIDERS = new Set([
  'deepseek/v4',
  'kimi/k2',
  'qwen/thinking',
  'newstrict/provider', // Add here
]);

Manual Cache Lookup in Custom Handlers

For debugging or custom logic, import the lookup utilities directly:

import { requiresReasoningReplay, lookupReasoning } from '@/open-sse/services/reasoningCache';

async function translateRequest(body, provider, model) {
  if (requiresReasoningReplay({ provider, model })) {
    for (const msg of body.messages) {
      if (msg.role === 'assistant' && msg.tool_calls) {
        for (const tc of msg.tool_calls) {
          const cached = await lookupReasoning({
            provider,
            model,
            toolCallId: tc.id,
          });
          if (cached) {
            tc.reasoning_content = cached;
          }
        }
      }
    }
  }
}

Summary

• OmniRoute uses a three-layer architecture (in-memory LRU, SQLite, cleanup job) to manage reasoning content for strict providers.
• cacheReasoningFromAssistantMessage captures reasoning from assistant responses in open-sse/handlers/chatCore.ts.
• requiresReasoningReplay and lookupReasoning automatically inject cached content into subsequent requests to prevent 400 errors.
• Entries persist for 24 hours by default, with a configurable cleanup interval (default 30 minutes).
• The cache exposes REST endpoints at /api/cache/reasoning for inspection and manual purging.

Frequently Asked Questions

What happens if the reasoning cache misses for a strict provider?

If lookupReasoning returns null for a strict provider, OmniRoute injects an empty string ("") into the reasoning_content field rather than omitting it. This prevents the upstream provider from returning a 400 "Param Incorrect" error, though the conversation may lack historical reasoning context for that specific tool call.

How long does OmniRoute retain reasoning content?

By default, reasoning cache entries expire after 24 hours. This TTL is configurable in the database schema, and a background cleanup job runs every 30 minutes (configurable via OMNIROUTE_REASONING_CACHE_CLEANUP_INTERVAL_MS) to remove expired rows from both SQLite and the in-memory store.

Which providers require reasoning replay in OmniRoute?

The current implementation maintains hard-coded allow-lists in open-sse/services/reasoningCache.ts including models like DeepSeek V4, Kimi K2, and Qwen-Thinking. The requiresReasoningReplay function checks against these sets to determine if a provider mandates the reasoning_content echo behavior.

Can I manually inspect or clear the reasoning cache?

Yes. OmniRoute exposes REST endpoints at /api/cache/reasoning (defined in src/app/api/cache/reasoning/route.ts). Send a GET request to retrieve hit/miss statistics and recent entries, or a DELETE request with optional provider or toolCallId filters to clear specific cached reasoning blocks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}