How OmniRoute's Semantic Cache Improves Response Times and Exposes Cache Headers

TLDR: OmniRoute eliminates upstream latency for deterministic LLM requests by storing responses in a SQLite semantic cache keyed by cryptographic signatures, serving cached results with X-OmniRoute-Cache-Hit: true headers in under 10ms while bypassing provider costs entirely.

OmniRoute is an open-source AI request router that reduces infrastructure costs through intelligent caching. Its semantic cache intercepts eligible requests before they reach upstream providers, storing deterministic completions in a local SQLite database for instant retrieval on subsequent identical calls.

How Semantic Caching Works

The semantic cache operates by generating unique signatures for deterministic requests and storing responses in the semantic_cache SQLite table. This process occurs entirely before any network I/O to upstream providers.

Request Eligibility and Signature Generation

Only deterministic requests qualify for caching. OmniRoute checks for temperature: 0, top_p: 1, identical model names, and optional API-key scoping. When a request meets these criteria, the system generates a cryptographic signature in src/lib/semanticCache.ts using the generateSignature function:

// Components hashed for the signature
const signature = generateSignature({
  model,
  prompt,      // Full prompt content
  temperature: 0,
  top_p: 1,
  apiKeyId     // Optional scoping
});

This signature serves as the primary key for cache lookups in src/lib/db/semanticCache.ts.

Cache Lookup: Hit vs. Miss

The core chat handler invokes checkSemanticCache() to query the database via getCachedResponse:

• Cache HIT: The stored response returns immediately from the local SQLite database, bypassing the upstream provider entirely. The system streams the result as Server-Sent Events (SSE) if requested.
• Cache MISS: The request proceeds to the configured LLM provider. After receiving the response, OmniRoute stores the result in the semantic_cache table for future lookups.

This lookup logic resides in open-sse/handlers/chatCore/semanticCache.ts, ensuring zero network overhead when cached content is available.

Cache Headers and Observability

OmniRoute attaches detailed metadata headers to every response, enabling downstream services to track cache efficiency and cost savings.

Understanding X-OmniRoute-Cache-Hit

When a response is served, the domain layer function attachOmniRouteMetaHeaders in src/domain/omnirouteResponseMeta.ts sets the header using the OMNIROUTE_RESPONSE_HEADERS.cacheHit constant defined in src/shared/constants/headers.ts:

• Hit: X-OmniRoute-Cache-Hit: true (and generic X-OmniRoute-Cache: HIT)
• Miss: X-OmniRoute-Cache-Hit: false

These headers allow client applications and monitoring dashboards to distinguish between cached and fresh responses without parsing the response body.

Cost and Latency Metadata

For cached responses, the cost-saved header reflects the would-be cost of the upstream call, while response-cost reports zero since no provider was invoked. Fresh requests populate actual token usage and latency metrics. This metadata is injected by attachOmniRouteMetaHeaders and provides complete observability for billing and performance analysis.

Enabling and Configuring the Cache

Administrators can toggle semantic caching at runtime via the settings API endpoint at src/app/api/settings/cache-config/route.ts:

// Enable semantic caching
await fetch('/api/settings/cache-config', {
  method: 'PATCH',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ semanticCacheEnabled: true })
});

Once enabled, the cache applies automatically to all qualifying requests without requiring client-side changes.

Client-Side Integration

Client applications can detect cache hits to optimize billing and UI behavior:

const resp = await fetch('/v1/chat/completions', { 
  method: 'POST', 
  body: JSON.stringify(payload) 
});

const cacheHit = resp.headers.get('X-OmniRoute-Cache-Hit'); // "true" or "false"

if (cacheHit === 'true') {
  // No provider cost incurred for this request
  console.log('Served from semantic cache');
}

Performance Impact

Because the cache lookup executes before any network I/O to LLM providers, a cache hit eliminates upstream latency entirely. While uncached requests to external providers typically measure in hundreds of milliseconds, cached responses return in under 10 milliseconds from the local SQLite database. This architecture significantly reduces API costs for repetitive deterministic workloads while maintaining sub-millisecond header injection overhead.

Summary

• Deterministic caching: OmniRoute caches only requests with temperature: 0 and top_p: 1 to ensure reproducible results.
• Cryptographic signatures: The generateSignature function in src/lib/semanticCache.ts creates unique hashes based on model, prompt, and parameters.
• Zero-cost hits: Cached responses bypass upstream providers, setting X-OmniRoute-Cache-Hit: true via OMNIROUTE_RESPONSE_HEADERS.cacheHit and zeroing out response costs.
• Observability headers: attachOmniRouteMetaHeaders in src/domain/omnirouteResponseMeta.ts injects latency, cost, and cache status metadata.
• Sub-10ms latency: Local SQLite lookups in src/lib/db/semanticCache.ts replace multi-hundred-millisecond upstream calls for cached content.

Frequently Asked Questions

How does OmniRoute determine if a request is cacheable?

OmniRoute validates deterministic parameters before caching. The request must use temperature: 0, top_p: 1, and a consistent model name. Additionally, the system considers optional API-key scoping to isolate caches between different clients. These checks occur in open-sse/handlers/chatCore/semanticCache.ts before generating the cryptographic signature.

What database does OmniRoute use for the semantic cache?

OmniRoute stores cached responses in a SQLite database table named semantic_cache. The database layer in src/lib/db/semanticCache.ts handles CRUD operations, while cryptographic signatures serve as primary keys for O(1) lookups.

How can I monitor cache hit rates in my application?

Check the X-OmniRoute-Cache-Hit response header, which returns "true" for cached responses and "false" for upstream calls. For detailed metrics, also inspect X-OmniRoute-Cache (set to HIT on cache hits) and the cost-savings headers injected by src/domain/omnirouteResponseMeta.ts.

Does enabling the semantic cache affect streaming responses?

No. When a cached response is served, OmniRoute streams the stored content as Server-Sent Events (SSE) if the client requested streaming, maintaining API compatibility while eliminating upstream latency. The X-OmniRoute-Cache-Hit: true header is present on both streamed and non-streamed cached responses.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}