# How OmniRoute's Semantic Cache Works: Implementation Guide for LLM Response Caching

> Learn how OmniRoute's semantic cache speeds up LLM responses with a two-tier system. This guide details its implementation for reduced latency and cost.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-09-01

---

**OmniRoute's semantic cache is a two-tier storage system that caches deterministic LLM responses (temperature = 0) to reduce latency and cost, using SHA-256 request signatures for cache keys with LRU memory and SQLite persistence layers.**

The **semantic cache** in OmniRoute is designed specifically for LLM workloads where identical prompts should yield identical outputs. This deep dive examines the complete architecture as implemented in `diegosouzapw/OmniRoute`, covering signature generation, storage mechanics, cache policies, and practical integration patterns.

## Cache Architecture and Design Principles

OmniRoute's semantic cache enforces **determinism as a prerequisite for caching**. Only requests with `temperature: 0` qualify—this prevents caching stochastic completions that would vary between identical prompts. The cache operates as a **read-through, write-through** layer integrated directly into the chat request pipeline.

### Two-Tier Storage System

The cache uses complementary storage layers defined in [`src/lib/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/semanticCache.ts):

- **In-memory LRU**: Fast, bounded by `SEMANTIC_CACHE_MAX_SIZE` (entry count) and `SEMANTIC_CACHE_MAX_BYTES` (memory). Initialized at [lines 10-17](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L10-L17).
- **SQLite table `semantic_cache`**: Persistent storage across restarts, with CRUD operations at [lines 62-88](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L62-L88).

This design optimizes for the Pareto distribution of cache access patterns: hot requests hit memory, while the SQLite backing ensures durability and larger capacity.

## Signature Generation: The Cache Key

Every cacheable request is identified by a **deterministic SHA-256 signature** generated by `generateSignature()` at [lines 49-71](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L49-L71). The signature incorporates:

- Model name (e.g., `"gpt-4o-mini"`)
- Normalized message array (order and content)
- `temperature` (must equal 0)
- `top_p` value
- Optional API-key prefix for **multi-tenant isolation**

```typescript
// Generate a cache signature for deterministic requests
import { generateSignature } from "@/lib/semanticCache";

const signature = generateSignature(
  "gpt-4o-mini",
  [{ role: "user", content: "Explain quantum tunneling." }],
  0,              // temperature: REQUIRED to be exactly 0
  1,              // top_p
  "apiKey-12345"  // optional: isolates cache by API key
);

```

The SHA-256 hash ensures collision-resistant, fixed-length keys suitable for both LRU and SQLite indexing.

## Cache Lookup and Write Flows

### Read Path: LRU-First with SQLite Fallback

`getCachedResponse()` ([lines 6-14](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L6-L14)) implements a **promotion-aware lookup**:

1. Check in-memory LRU first
2. On miss, query SQLite `semantic_cache` table
3. Promote SQLite hits back into LRU
4. Update hit metrics

```typescript
// Cache lookup pattern in request handlers
import { getCachedResponse } from "@/lib/semanticCache";

const cached = await getCachedResponse(signature);
if (cached) {
  // Return cached JSON directly—bypass provider entirely
  return Response.json(cached.response);
}

```

### Write Path: Dual-Layer Persistence

After receiving a provider response, `setCachedResponse()` ([lines 63-71](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L63-L71)) writes to both layers:

```typescript
// Store fresh response after provider completes
import { setCachedResponse } from "@/lib/semanticCache";

await setCachedResponse(
  signature,
  "gpt-4o-mini",
  providerResponse,      // Full LLM JSON payload
  estimatedTokensSaved,  // For cost analytics
  1800000               // TTL: 30 minutes (default from SEMANTIC_CACHE_TTL_MS)
);

```

The default **30-minute TTL** balances freshness with hit rate—adjustable per-write or via environment variable.

## Cacheability Rules and Safety

OmniRoute strictly validates cache eligibility through `isCacheableForRead()` and `isCacheableForWrite()` ([lines 87-100](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L87-L100)). A request is **cacheable only when**:

- `X-OmniRoute-No-Cache: true` header is **absent**
- Request body explicitly sets `temperature: 0`

This dual-gate prevents accidental caching of non-deterministic outputs and provides escape hatches for cache-bypass scenarios.

## Cache Invalidation Strategies

OmniRoute provides granular invalidation helpers at [lines 92-99](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L92-L99) and related functions:

```typescript
// Invalidate all entries for a specific model (e.g., after upgrade)
import { invalidateByModel } from "@/lib/semanticCache";

const removed = await invalidateByModel("gpt-4o-mini");
console.info(`Invalidated ${removed} cached rows`);

// Additional helpers (by signature, age, or complete purge)
// invalidateBySignature(signature)
// invalidateOlderThan(timestampMs)
// invalidateAll()

```

Invalidation clears LRU entries **before** SQLite deletion to maintain consistency.

## Metrics and Observability

The semantic cache records operational data in two dimensions:

- **`cache_metrics` table**: Persistent storage of hits, misses, and tokens saved
- **LRU runtime stats**: Exposed via `/api/cache/stats` endpoint

Use `getCacheStats()` ([lines 66-78](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/semanticCache.ts#L66-L78)) and `incrementMetric()` for custom instrumentation.

## Integration in Request Pipeline

The semantic cache hooks into the chat core at two integration points in `open-sse/handlers/chatCore/`:

| File | Responsibility |
|------|---------------|
| [[`semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/semanticCache.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/open-sse/handlers/chatCore/semanticCache.ts) | Pre-request lookup and post-response cache decision |
| [[`semanticCacheStore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/semanticCacheStore.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/open-sse/handlers/chatCore/semanticCacheStore.ts) | Buffers and stores complete streaming responses |

This architecture keeps cache logic decoupled from provider-specific implementations while ensuring zero overhead for non-cacheable requests.

## Summary

- **Determinism requirement**: Only `temperature: 0` requests are cached, enforced by `isCacheableForRead/Write`
- **SHA-256 signatures**: `generateSignature()` creates collision-resistant, normalized keys including optional API-key isolation
- **Two-tier storage**: LRU memory for speed, SQLite `semantic_cache` table for persistence
- **30-minute default TTL**: Configurable via `SEMANTIC_CACHE_TTL_MS` or per-write override
- **Promotion on miss**: SQLite hits are re-inserted into LRU to optimize subsequent accesses
- **Multi-model invalidation**: `invalidateByModel()` and helpers support operational cache management
- **Integrated metrics**: Track hits, misses, and cost savings via `cache_metrics` and runtime stats

## Frequently Asked Questions

### How does OmniRoute ensure cached responses are deterministic?

OmniRoute requires **explicit `temperature: 0`** in the request body and validates absence of the `X-OmniRoute-No-Cache: true` header. The `isCacheableForRead()` and `isCacheableForWrite()` functions in [`src/lib/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/semanticCache.ts) enforce these rules before any cache operation.

### What happens when the in-memory LRU is full?

The LRU evicts least-recently-used entries based on configured `SEMANTIC_CACHE_MAX_SIZE` (entry count) and `SEMANTIC_CACHE_MAX_BYTES` (memory). Evicted entries remain in SQLite and can be promoted back to LRU on subsequent hits.

### How does multi-tenant API key isolation work?

The optional `apiKeyPrefix` parameter in `generateSignature()` prepends a tenant identifier to the hash input. This creates **namespace-isolated cache keys**, preventing cross-tenant data leakage while allowing shared infrastructure.

### Can I disable or bypass the semantic cache for specific requests?

Yes. Add the header `X-OmniRoute-No-Cache: true` to any request, or ensure `temperature` is not exactly 0. The cache integration checks these conditions before lookup and will always forward non-cacheable requests to the provider.