# How OmniRoute Semantic Caching Detects Cache Hits and Misses

> Learn how OmniRoute semantic caching detects hits and misses using SHA-256 signatures and a two-tier LRU/SQLite cache. Optimize LLM responses and track token savings.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-08

---

**OmniRoute uses a deterministic two-tier semantic cache that generates SHA-256 signatures from normalized requests to store LLM responses, checking an in-memory LRU cache before falling back to SQLite while tracking hits, misses, and token savings via dedicated metrics.**

OmniRoute is an open-source LLM routing layer that implements semantic caching to reduce costs and latency for deterministic prompts. The semantic caching system, implemented primarily in [`src/lib/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/semanticCache.ts), creates unique signatures for requests with `temperature` set to zero and maintains a two-tier storage architecture. This article explains exactly how the system detects cache hits and misses based on the diegosouzapw/OmniRoute source code.

## Generating Deterministic Cache Signatures

Every cache operation starts with a **cache signature** that uniquely identifies a request. The `generateSignature` function in [`src/lib/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/semanticCache.ts) (lines 19-41) creates a SHA-256 hash from:

- The **model name** (e.g., "gpt-4o-mini")
- The **normalized conversation** array
- The `temperature` value (must be 0 for caching)
- The `top_p` parameter

To prevent cross-user collisions, the system supports **API-key isolation**. When an `apiKeyId` is provided, it is prefixed as plain text to the hash before storage (lines 33-40). This ensures that identical prompts from different users remain isolated while maintaining deterministic lookups within a single user's context.

```typescript
// Build a deterministic cache signature for a request
import { generateSignature } from "@/lib/semanticCache";

const signature = generateSignature(
  "gpt-4o-mini",               // model
  [{ role: "user", content: "What is the capital of France?" }], // conversation
  0,                            // temperature (must be zero)
  1,                            // top_p
  apiKeyId                      // optional API-key isolation
);

```

## Two-Tier Storage Architecture

OmniRoute implements a **two-tier storage system** that balances speed with persistence.

### In-Memory LRU Cache

The **memory tier** uses an `LRUCache` instance created on first use by `getMemoryCache()` (lines 94-106). This in-process cache provides sub-millisecond lookups for hot data and is ideal for high-frequency repeated prompts. The cache is maintained within the Node.js process and is cleared on restart.

### Persistent SQLite Layer

The **SQLite tier** stores data in a `semantic_cache` table that survives process restarts. The cache accesses this layer via the shared database singleton `getDbInstance()` defined in [`src/lib/db/core.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/core.ts). Each row includes an `expires_at` timestamp that enables automatic eviction of stale entries without requiring manual cleanup.

## Detecting Cache Hits and Misses

The `getCachedResponse(signature)` function orchestrates hit and miss detection for every eligible request.

### Hit Detection Flow

When a request arrives, the system executes a tiered lookup:

1. **Memory check**: The function first queries the LRU cache. If found, it calls `incrementMetric("hits")` and immediately returns the stored JSON response (lines 77-83).
2. **Database fallback**: If the memory tier misses, the system queries the SQLite `semantic_cache` table for a row matching the signature where `expires_at` is in the future (lines 86-108).
3. **Promotion**: When the database contains a valid entry, the response is parsed, promoted to the memory cache for subsequent requests, and the hit metrics are incremented before returning the data.

```typescript
// Attempt a cache read (hit → response, miss → null)
import { getCachedResponse, isCacheableForRead } from "@/lib/semanticCache";

if (isCacheableForRead(req.body, req.headers)) {
  const cached = getCachedResponse(signature);
  if (cached) {
    // Cache hit – immediately return the stored JSON
    return res.json(cached);
  }
}

```

### Miss Detection Logic

A **cache miss** occurs only when both tiers return null or invalid data. In this case, `getCachedResponse` calls `incrementMetric("misses")` (lines 119-122) and returns `null`, signaling the router to proceed with a fresh LLM call. The miss is recorded before the outgoing request is initiated, ensuring accurate latency tracking.

```typescript
// After a fresh LLM call, store the response (if cache-able)
import { setCachedResponse, isCacheableForWrite } from "@/lib/semanticCache";

if (isCacheableForWrite(req.body, req.headers)) {
  // `llmResponse` is the JSON payload returned by the provider
  const tokensSaved = estimateTokensSaved(req.body, llmResponse);
  setCachedResponse(signature, "gpt-4o-mini", llmResponse, tokensSaved);
}

```

## Cache Eligibility Rules

Not all requests participate in semantic caching. The system enforces strict eligibility criteria through two helper functions:

- **`isCacheableForRead(body, headers)`**: Returns `true` only if `temperature === 0` and the request does not contain the header `X-OmniRoute-No-Cache: true` (lines 57-63).
- **`isCacheableForWrite(body, headers)`**: Applies identical rules after a streaming or non-streaming response completes to determine if the result should be persisted (lines 71-77).

These guards ensure that non-deterministic requests (temperature > 0) or explicitly bypassed calls never pollute the cache, maintaining the integrity of stored responses.

## Metrics and Observability

The cache tracks performance via a lightweight `cache_metrics` table. The helper functions `incrementMetric` and `getMetricValue` (lines 52-71) update and read counters for total **hits**, **misses**, and **tokens saved**. You can retrieve comprehensive statistics using `getCacheStats()`, which returns memory-cache size, database row counts, and calculated hit rates.

```typescript
// Retrieve cache statistics for monitoring dashboards
import { getCacheStats } from "@/lib/semanticCache";

const stats = getCacheStats();
console.log("Semantic cache:", stats);

```

## Summary

- OmniRoute semantic caching uses **SHA-256 signatures** generated from model name, conversation content, temperature, and top_p to create deterministic cache keys.
- A **two-tier system** combines an in-memory LRU cache for speed with a persistent SQLite table for durability across restarts.
- **Hit detection** checks the memory tier first, falls back to SQLite with expiration validation, and promotes database hits to memory.
- **Miss detection** increments the miss counter and returns null only when both storage tiers fail to produce a valid entry.
- **Eligibility rules** restrict caching to requests with `temperature = 0` that do not include the `X-OmniRoute-No-Cache` header.
- Built-in **metrics tracking** monitors hits, misses, and token savings via the `cache_metrics` table.

## Frequently Asked Questions

### What makes a request eligible for OmniRoute's semantic cache?

A request must have `temperature` set to exactly `0` and must not include the header `X-OmniRoute-No-Cache: true`. These rules are enforced by `isCacheableForRead()` and `isCacheableForWrite()` in [`src/lib/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/semanticCache.ts). Non-deterministic requests with temperature greater than zero are always excluded to prevent storing variable responses.

### How does OmniRoute prevent cache collisions between different API keys?

The system supports optional **API-key isolation** by prefixing an `apiKeyId` to the cache signature before hashing (lines 33-40). When provided, this ID ensures that identical prompts from different users generate different cache keys, preventing cross-tenant data leakage while still allowing shared caching within a single user's scope.

### What happens when the in-memory cache misses but the SQLite cache hits?

When the LRU cache misses but SQLite contains a valid unexpired entry, the system parses the stored response, increments the hit metric, **promotes the entry to the memory cache** for subsequent requests, and returns the data to the client. This promotion strategy warms the fast tier with recently accessed data.

### How can I monitor semantic cache performance in OmniRoute?

Import `getCacheStats()` from `@/lib/semanticCache` to retrieve real-time statistics including memory cache size, database row count, hit rate percentage, and total tokens saved. The underlying `cache_metrics` table tracks discrete counters for hits and misses that persist across restarts, enabling long-term trend analysis.