# How OmniRoute's Semantic Cache Improves Response Times and Exposes Cache Headers

> Discover how OmniRoute's semantic cache slashes LLM response times by serving cached results under 10ms with X-OmniRoute-Cache-Hit headers, eliminating upstream latency and provider costs.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-07-25

---

**TLDR:** OmniRoute eliminates upstream latency for deterministic LLM requests by storing responses in a SQLite semantic cache keyed by cryptographic signatures, serving cached results with `X-OmniRoute-Cache-Hit: true` headers in under 10ms while bypassing provider costs entirely.

OmniRoute is an open-source AI request router that reduces infrastructure costs through intelligent caching. Its **semantic cache** intercepts eligible requests before they reach upstream providers, storing deterministic completions in a local SQLite database for instant retrieval on subsequent identical calls.

## How Semantic Caching Works

The semantic cache operates by generating unique signatures for deterministic requests and storing responses in the `semantic_cache` SQLite table. This process occurs entirely before any network I/O to upstream providers.

### Request Eligibility and Signature Generation

Only deterministic requests qualify for caching. OmniRoute checks for `temperature: 0`, `top_p: 1`, identical model names, and optional API-key scoping. When a request meets these criteria, the system generates a cryptographic **signature** in [`src/lib/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/semanticCache.ts) using the `generateSignature` function:

```typescript
// Components hashed for the signature
const signature = generateSignature({
  model,
  prompt,      // Full prompt content
  temperature: 0,
  top_p: 1,
  apiKeyId     // Optional scoping
});

```

This signature serves as the primary key for cache lookups in [`src/lib/db/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/semanticCache.ts).

### Cache Lookup: Hit vs. Miss

The core chat handler invokes `checkSemanticCache()` to query the database via `getCachedResponse`:

- **Cache HIT:** The stored response returns immediately from the local SQLite database, bypassing the upstream provider entirely. The system streams the result as Server-Sent Events (SSE) if requested.
- **Cache MISS:** The request proceeds to the configured LLM provider. After receiving the response, OmniRoute stores the result in the `semantic_cache` table for future lookups.

This lookup logic resides in [`open-sse/handlers/chatCore/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore/semanticCache.ts), ensuring zero network overhead when cached content is available.

## Cache Headers and Observability

OmniRoute attaches detailed metadata headers to every response, enabling downstream services to track cache efficiency and cost savings.

### Understanding X-OmniRoute-Cache-Hit

When a response is served, the domain layer function `attachOmniRouteMetaHeaders` in [`src/domain/omnirouteResponseMeta.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/omnirouteResponseMeta.ts) sets the header using the `OMNIROUTE_RESPONSE_HEADERS.cacheHit` constant defined in [`src/shared/constants/headers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/headers.ts):

- **Hit:** `X-OmniRoute-Cache-Hit: true` (and generic `X-OmniRoute-Cache: HIT`)
- **Miss:** `X-OmniRoute-Cache-Hit: false`

These headers allow client applications and monitoring dashboards to distinguish between cached and fresh responses without parsing the response body.

### Cost and Latency Metadata

For cached responses, the `cost-saved` header reflects the would-be cost of the upstream call, while `response-cost` reports zero since no provider was invoked. Fresh requests populate actual token usage and latency metrics. This metadata is injected by `attachOmniRouteMetaHeaders` and provides complete observability for billing and performance analysis.

## Enabling and Configuring the Cache

Administrators can toggle semantic caching at runtime via the settings API endpoint at [`src/app/api/settings/cache-config/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/settings/cache-config/route.ts):

```typescript
// Enable semantic caching
await fetch('/api/settings/cache-config', {
  method: 'PATCH',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ semanticCacheEnabled: true })
});

```

Once enabled, the cache applies automatically to all qualifying requests without requiring client-side changes.

## Client-Side Integration

Client applications can detect cache hits to optimize billing and UI behavior:

```typescript
const resp = await fetch('/v1/chat/completions', { 
  method: 'POST', 
  body: JSON.stringify(payload) 
});

const cacheHit = resp.headers.get('X-OmniRoute-Cache-Hit'); // "true" or "false"

if (cacheHit === 'true') {
  // No provider cost incurred for this request
  console.log('Served from semantic cache');
}

```

## Performance Impact

Because the cache lookup executes before any network I/O to LLM providers, a cache hit eliminates upstream latency entirely. While uncached requests to external providers typically measure in hundreds of milliseconds, cached responses return in under 10 milliseconds from the local SQLite database. This architecture significantly reduces API costs for repetitive deterministic workloads while maintaining sub-millisecond header injection overhead.

## Summary

- **Deterministic caching:** OmniRoute caches only requests with `temperature: 0` and `top_p: 1` to ensure reproducible results.
- **Cryptographic signatures:** The `generateSignature` function in [`src/lib/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/semanticCache.ts) creates unique hashes based on model, prompt, and parameters.
- **Zero-cost hits:** Cached responses bypass upstream providers, setting `X-OmniRoute-Cache-Hit: true` via `OMNIROUTE_RESPONSE_HEADERS.cacheHit` and zeroing out response costs.
- **Observability headers:** `attachOmniRouteMetaHeaders` in [`src/domain/omnirouteResponseMeta.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/omnirouteResponseMeta.ts) injects latency, cost, and cache status metadata.
- **Sub-10ms latency:** Local SQLite lookups in [`src/lib/db/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/semanticCache.ts) replace multi-hundred-millisecond upstream calls for cached content.

## Frequently Asked Questions

### How does OmniRoute determine if a request is cacheable?

OmniRoute validates deterministic parameters before caching. The request must use `temperature: 0`, `top_p: 1`, and a consistent model name. Additionally, the system considers optional API-key scoping to isolate caches between different clients. These checks occur in [`open-sse/handlers/chatCore/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore/semanticCache.ts) before generating the cryptographic signature.

### What database does OmniRoute use for the semantic cache?

OmniRoute stores cached responses in a **SQLite** database table named `semantic_cache`. The database layer in [`src/lib/db/semanticCache.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/semanticCache.ts) handles CRUD operations, while cryptographic signatures serve as primary keys for O(1) lookups.

### How can I monitor cache hit rates in my application?

Check the `X-OmniRoute-Cache-Hit` response header, which returns `"true"` for cached responses and `"false"` for upstream calls. For detailed metrics, also inspect `X-OmniRoute-Cache` (set to `HIT` on cache hits) and the cost-savings headers injected by [`src/domain/omnirouteResponseMeta.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/omnirouteResponseMeta.ts).

### Does enabling the semantic cache affect streaming responses?

No. When a cached response is served, OmniRoute streams the stored content as Server-Sent Events (SSE) if the client requested streaming, maintaining API compatibility while eliminating upstream latency. The `X-OmniRoute-Cache-Hit: true` header is present on both streamed and non-streamed cached responses.