How OmniRoute Semantic Caching Detects Cache Hits and Misses

OmniRoute uses a deterministic two-tier semantic cache that generates SHA-256 signatures from normalized requests to store LLM responses, checking an in-memory LRU cache before falling back to SQLite while tracking hits, misses, and token savings via dedicated metrics.

OmniRoute is an open-source LLM routing layer that implements semantic caching to reduce costs and latency for deterministic prompts. The semantic caching system, implemented primarily in src/lib/semanticCache.ts, creates unique signatures for requests with temperature set to zero and maintains a two-tier storage architecture. This article explains exactly how the system detects cache hits and misses based on the diegosouzapw/OmniRoute source code.

Generating Deterministic Cache Signatures

Every cache operation starts with a cache signature that uniquely identifies a request. The generateSignature function in src/lib/semanticCache.ts (lines 19-41) creates a SHA-256 hash from:

• The model name (e.g., "gpt-4o-mini")
• The normalized conversation array
• The temperature value (must be 0 for caching)
• The top_p parameter

To prevent cross-user collisions, the system supports API-key isolation. When an apiKeyId is provided, it is prefixed as plain text to the hash before storage (lines 33-40). This ensures that identical prompts from different users remain isolated while maintaining deterministic lookups within a single user's context.

// Build a deterministic cache signature for a request
import { generateSignature } from "@/lib/semanticCache";

const signature = generateSignature(
  "gpt-4o-mini",               // model
  [{ role: "user", content: "What is the capital of France?" }], // conversation
  0,                            // temperature (must be zero)
  1,                            // top_p
  apiKeyId                      // optional API-key isolation
);

Two-Tier Storage Architecture

OmniRoute implements a two-tier storage system that balances speed with persistence.

In-Memory LRU Cache

The memory tier uses an LRUCache instance created on first use by getMemoryCache() (lines 94-106). This in-process cache provides sub-millisecond lookups for hot data and is ideal for high-frequency repeated prompts. The cache is maintained within the Node.js process and is cleared on restart.

Persistent SQLite Layer

The SQLite tier stores data in a semantic_cache table that survives process restarts. The cache accesses this layer via the shared database singleton getDbInstance() defined in src/lib/db/core.ts. Each row includes an expires_at timestamp that enables automatic eviction of stale entries without requiring manual cleanup.

Detecting Cache Hits and Misses

The getCachedResponse(signature) function orchestrates hit and miss detection for every eligible request.

Hit Detection Flow

When a request arrives, the system executes a tiered lookup:

1. Memory check: The function first queries the LRU cache. If found, it calls incrementMetric("hits") and immediately returns the stored JSON response (lines 77-83).
2. Database fallback: If the memory tier misses, the system queries the SQLite semantic_cache table for a row matching the signature where expires_at is in the future (lines 86-108).
3. Promotion: When the database contains a valid entry, the response is parsed, promoted to the memory cache for subsequent requests, and the hit metrics are incremented before returning the data.

// Attempt a cache read (hit → response, miss → null)
import { getCachedResponse, isCacheableForRead } from "@/lib/semanticCache";

if (isCacheableForRead(req.body, req.headers)) {
  const cached = getCachedResponse(signature);
  if (cached) {
    // Cache hit – immediately return the stored JSON
    return res.json(cached);
  }
}

Miss Detection Logic

A cache miss occurs only when both tiers return null or invalid data. In this case, getCachedResponse calls incrementMetric("misses") (lines 119-122) and returns null, signaling the router to proceed with a fresh LLM call. The miss is recorded before the outgoing request is initiated, ensuring accurate latency tracking.

// After a fresh LLM call, store the response (if cache-able)
import { setCachedResponse, isCacheableForWrite } from "@/lib/semanticCache";

if (isCacheableForWrite(req.body, req.headers)) {
  // `llmResponse` is the JSON payload returned by the provider
  const tokensSaved = estimateTokensSaved(req.body, llmResponse);
  setCachedResponse(signature, "gpt-4o-mini", llmResponse, tokensSaved);
}

Cache Eligibility Rules

Not all requests participate in semantic caching. The system enforces strict eligibility criteria through two helper functions:

• isCacheableForRead(body, headers): Returns true only if temperature === 0 and the request does not contain the header X-OmniRoute-No-Cache: true (lines 57-63).
• isCacheableForWrite(body, headers): Applies identical rules after a streaming or non-streaming response completes to determine if the result should be persisted (lines 71-77).

These guards ensure that non-deterministic requests (temperature > 0) or explicitly bypassed calls never pollute the cache, maintaining the integrity of stored responses.

Metrics and Observability

The cache tracks performance via a lightweight cache_metrics table. The helper functions incrementMetric and getMetricValue (lines 52-71) update and read counters for total hits, misses, and tokens saved. You can retrieve comprehensive statistics using getCacheStats(), which returns memory-cache size, database row counts, and calculated hit rates.

// Retrieve cache statistics for monitoring dashboards
import { getCacheStats } from "@/lib/semanticCache";

const stats = getCacheStats();
console.log("Semantic cache:", stats);

Summary

• OmniRoute semantic caching uses SHA-256 signatures generated from model name, conversation content, temperature, and top_p to create deterministic cache keys.
• A two-tier system combines an in-memory LRU cache for speed with a persistent SQLite table for durability across restarts.
• Hit detection checks the memory tier first, falls back to SQLite with expiration validation, and promotes database hits to memory.
• Miss detection increments the miss counter and returns null only when both storage tiers fail to produce a valid entry.
• Eligibility rules restrict caching to requests with temperature = 0 that do not include the X-OmniRoute-No-Cache header.
• Built-in metrics tracking monitors hits, misses, and token savings via the cache_metrics table.

Frequently Asked Questions

What makes a request eligible for OmniRoute's semantic cache?

A request must have temperature set to exactly 0 and must not include the header X-OmniRoute-No-Cache: true. These rules are enforced by isCacheableForRead() and isCacheableForWrite() in src/lib/semanticCache.ts. Non-deterministic requests with temperature greater than zero are always excluded to prevent storing variable responses.

How does OmniRoute prevent cache collisions between different API keys?

The system supports optional API-key isolation by prefixing an apiKeyId to the cache signature before hashing (lines 33-40). When provided, this ID ensures that identical prompts from different users generate different cache keys, preventing cross-tenant data leakage while still allowing shared caching within a single user's scope.

What happens when the in-memory cache misses but the SQLite cache hits?

When the LRU cache misses but SQLite contains a valid unexpired entry, the system parses the stored response, increments the hit metric, promotes the entry to the memory cache for subsequent requests, and returns the data to the client. This promotion strategy warms the fast tier with recently accessed data.

How can I monitor semantic cache performance in OmniRoute?

Import getCacheStats() from @/lib/semanticCache to retrieve real-time statistics including memory cache size, database row count, hit rate percentage, and total tokens saved. The underlying cache_metrics table tracks discrete counters for hits and misses that persist across restarts, enabling long-term trend analysis.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}