How OmniRoute's Cache Check Works: Idempotency and Semantic Caching Explained

OmniRoute's cache check uses a deterministic SHA-256 signature to deduplicate in-flight requests for idempotency and stores deterministic responses (temperature=0) in a two-tier LRU and SQLite cache for semantic caching.

The diegosouzapw/OmniRoute repository implements a sophisticated cache check mechanism that prevents redundant API calls and reduces provider costs through intelligent request deduplication and response reuse. Operating within the chat completion pipeline, this system combines idempotency protection for concurrent identical requests with semantic caching for deterministic chat completions, utilizing a shared deterministic signature generated from essential request parameters.

The Deterministic Signature Foundation

Both the idempotency and semantic caching systems rely on a unified signature generation strategy implemented in src/lib/semanticCache.ts. The generateSignature function creates a SHA-256 hash from a normalized concatenation of the model name, normalized messages, temperature, top_p, and an optional API-key ID. This signature serves as the unique identifier for cache lookups and in-flight request deduplication, ensuring that semantically identical requests produce identical signatures regardless of metadata differences.

Idempotency Check: In-Flight Request Deduplication

The idempotency layer operates as the first phase of the request pipeline in open-sse/handlers/chatCore/idempotency.ts. Its primary function is to prevent race conditions and duplicate billing when multiple identical requests arrive simultaneously.

How the Idempotency Layer Works

When a request enters the system, the handler generates a signature and checks an in-memory map of in-flight requests. If a matching signature exists, the subsequent request waits for the original promise to resolve rather than initiating a new upstream provider call. Once the original request completes, the result propagates to all waiting callers. The implementation stores active request promises in the in-flight map and cleans them up upon completion, ensuring that simultaneous identical requests consume only a single upstream API quota.

Why Request Deduplication Matters

This mechanism prevents double billing, eliminates redundant load on provider APIs, and ensures consistency when clients retry requests due to network timeouts. According to the source code in open-sse/handlers/chatCore/idempotency.ts, the system guarantees that regardless of how many clients submit the same request during the processing window, only one upstream call executes while all callers receive the identical response.

Semantic Caching Strategy

The semantic cache, implemented primarily in src/lib/semanticCache.ts, stores deterministic responses for reuse across different request sessions. Unlike simple key-value caching, this system validates that responses are sufficiently deterministic to cache safely.

Cache Read Logic and Temperature Requirements

The isCacheableForRead function enforces strict eligibility criteria before consulting the cache:

1. Client Override: Rejects caching when the request includes the X-OmniRoute-No-Cache: true header
2. Determinism Check: Accepts only requests with explicit temperature: 0, as this is the only temperature value that guarantees deterministic output from language models
3. Lookup Chain: If cacheable, getCachedResponse queries the in-memory LRU cache (src/lib/cacheLayer.ts) first, then falls back to the persistent SQLite table semantic_cache via src/lib/db/semanticCache.ts

On a cache hit, the system immediately returns the stored response—formatted correctly for both streaming and non-streaming callers—and increments metrics including hits and tokens_saved.

Cache Write Logic and Persistence

After receiving an upstream response, the isCacheableForWrite function applies the same validation rules (temperature=0 and no X-OmniRoute-No-Cache header). When valid, setCachedResponse writes the response to both the LRU cache and the SQLite persistence layer, recording the signature, model, TTL, hit count, and estimated tokens saved. This two-tier approach ensures low-latency access for recent responses while maintaining durability across service restarts.

Cache Invalidation Mechanisms

The system provides several invalidation strategies to maintain freshness after model updates or configuration changes:

• invalidateByModel(model): Removes all cached entries for a specific model
• invalidateBySignature(signature): Targets a specific cached response
• invalidateStale(): Purges expired entries based on TTL
• `clearCache()**: Wipes the entire semantic cache

These helpers ensure that stale responses do not persist after provider model upgrades or system configuration changes.

The Request Pipeline Execution Flow

The OmniRoute cache check executes through a five-phase pipeline where ordering is critical for efficiency:

1. Idempotency Phase (open-sse/handlers/chatCore/idempotency.ts): Generate signature and check for in-flight duplicates
2. Semantic Cache Read (open-sse/handlers/chatCore/semanticCache.ts): Query cache if request is eligible (temperature=0, no cache header)
3. Upstream Execution: Call the LLM provider if cache misses and no in-flight request exists
4. Semantic Cache Write (open-sse/handlers/chatCore/semanticCache.ts): Store deterministic responses for future reuse
5. Response Delivery: Return cached or fresh response to the client

This ordering ensures that a request already being processed triggers the idempotency hit rather than a cache miss, preventing unnecessary database lookups and further reducing provider API calls.

Working with the Cache API

The following TypeScript examples demonstrate interacting with OmniRoute's semantic caching system:

import {
  generateSignature,
  isCacheableForRead,
  getCachedResponse,
  setCachedResponse,
  clearCache,
} from "@/src/lib/semanticCache";

// Build a deterministic signature for cache operations
const sig = generateSignature(
  "gpt-4o-mini",          // model
  [{ role: "user", content: "What is the capital of France?" }], // messages
  0,                      // temperature (must be 0 for caching)
  1,                      // top_p
  apiKeyId,               // optional per-key isolation
);

// Check cache eligibility and retrieve cached response
if (isCacheableForRead(requestBody, requestHeaders)) {
  const cached = getCachedResponse(sig);
  if (cached) {
    // Fast-path: return cached answer immediately
    return new Response(JSON.stringify(cached), { status: 200 });
  }
}

// ... execute upstream provider call if cache misses ...

// Store response if deterministic (temperature=0 and no cache bypass)
if (isCacheableForWrite(requestBody, responseHeaders)) {
  setCachedResponse(sig, "gpt-4o-mini", providerResponse, tokensSaved);
}

To invalidate the entire cache after a major model update:

import { clearCache } from "@/src/lib/semanticCache";

const removed = clearCache();
console.info(`Cleared ${removed} semantic-cache entries`);

Summary

• OmniRoute's cache check combines idempotency and semantic caching using a shared SHA-256 signature generated from normalized request parameters
• The idempotency layer in open-sse/handlers/chatCore/idempotency.ts prevents duplicate in-flight requests by maintaining a promise map, ensuring only one upstream call executes for simultaneous identical requests
• Semantic caching requires temperature: 0 explicitly and stores responses in a two-tier system: in-memory LRU (src/lib/cacheLayer.ts) and persistent SQLite (src/lib/db/semanticCache.ts)
• Cache invalidation supports model-specific purging, signature-based removal, and complete cache clearing via invalidateByModel, invalidateBySignature, and clearCache
• The pipeline ordering places idempotency checks before cache reads, optimizing for the common case of duplicate requests during high-concurrency scenarios

Frequently Asked Questions

What makes a request cacheable in OmniRoute?

A request is cacheable only when it includes an explicit temperature: 0 parameter and does not contain the X-OmniRoute-No-Cache: true header. The temperature requirement ensures deterministic outputs, while the header provides client-side cache bypass capability. Both rules are enforced by isCacheableForRead and isCacheableForWrite in src/lib/semanticCache.ts.

How does OmniRoute prevent duplicate API calls?

OmniRoute prevents duplicate calls through the idempotency check in open-sse/handlers/chatCore/idempotency.ts. When multiple requests with identical signatures arrive while one is processing, subsequent requests wait for the original promise instead of initiating new upstream calls. This in-flight request deduplication eliminates race conditions and prevents double billing.

Where does OmniRoute store cached responses?

Cached responses reside in a two-tier storage system. The primary layer is an in-memory LRU cache implemented in src/lib/cacheLayer.ts for low-latency access. The secondary layer persists data in a SQLite database table semantic_cache managed by src/lib/db/semanticCache.ts, ensuring durability across service restarts while tracking TTL and hit metrics.

How do I invalidate the semantic cache?

Use the invalidation helpers exported from src/lib/semanticCache.ts: call invalidateByModel("model-name") to clear entries for a specific model, invalidateBySignature(sig) to remove a specific cached response, or clearCache() to wipe the entire cache. These functions update both the LRU and SQLite layers synchronously.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}