How the Idempotency Cache Works in OmniRoute: In-Memory Deduplication for Reliable API Requests

OmniRoute uses a lightweight in-memory Map-based cache to deduplicate requests by their Idempotency-Key header, returning cached responses for retries within a configurable time window.

The idempotency cache in OmniRoute protects downstream providers from redundant calls and ensures exactly-once semantics for safe retries. This mechanism is essential for handling network timeouts, client retries, and duplicate submissions without re-executing expensive operations like LLM inference or quota-tracked requests.

Core Architecture of the Idempotency Cache

The cache implementation spans two primary layers: a reusable library module and a handler-specific integration point. Together they provide a deterministic key extraction, lookup, and storage pipeline.

Key Extraction: getIdempotencyKey() in src/lib/idempotencyLayer.ts

The entry point for all idempotent requests is the getIdempotencyKey() function. This utility extracts the idempotency identifier from incoming headers, supporting both standard Headers objects and plain object maps for flexibility across different request types.

The function checks for two header variants in priority order:

1. idempotency-key — the standard header per RFC 7239 conventions
2. x-request-id — legacy fallback for backward compatibility

If neither header is present, the function returns null and idempotency is bypassed for that request.

Cache Storage: In-Memory Map with TTL

The underlying data structure is a simple Map<string, IdempotentEntry> stored in process memory. Each entry contains:

Property	Type	Purpose
status	number	HTTP status code to return
body	string	Serialized response body
expires	number	Timestamp when the entry becomes invalid

The default expiration window is DEFAULT_WINDOW_MS (5 minutes), configurable via the OMNIROUTE_IDEMPOTENCY_WINDOW_MS environment variable.

Lookup: checkIdempotency() Returns Cached Responses

The checkIdempotency(key) function performs a fast Map lookup with automatic expiration handling:

• Hit within window: Returns the full entry immediately
• Expired entry: Deletes the stale entry and returns null
• Miss: Returns null, allowing normal request processing

Storage: saveIdempotency() Captures Outgoing Responses

After successful request execution, the saveIdempotency() function persists the response. This ensures that any immediate retry—common when clients experience timeouts—hits the cache rather than re-triggering the operation.

Integration in Chat Core Handler: open-sse/handlers/chatCore/idempotency.ts

The chat processing pipeline wraps its execution logic with the idempotency layer. According to the OmniRoute source code, this handler orchestrates the full lifecycle:

1. Extract key from request headers
2. Check cache for existing response
3. Return cached result immediately on hit
4. Execute normal flow on miss
5. Store final response for future retries

This integration point is critical because chat requests often involve:

• Expensive LLM provider calls with per-token pricing
• Quota tracking that must not increment on duplicates
• Streaming responses that get buffered and cached as complete payloads

Complete Implementation Example

Cache Library: src/lib/idempotencyLayer.ts

// Core cache storage with TTL-based expiration
const CACHE = new Map<string, {
  status: number;
  body: string;
  expires: number;
}>();

const DEFAULT_WINDOW_MS = 5 * 60 * 1000; // 5 minutes

/**
 * Extract idempotency key from various header formats.
 * Supports both Headers API and plain objects.
 */
export function getIdempotencyKey(
  headers: Headers | Record<string, string>
): string | null {
  const get =
    typeof (headers as Headers).get === "function"
      ? (k: string) => (headers as Headers).get(k)
      : (k: string) => (headers as Record<string, string>)[k];

  return get("idempotency-key") ?? get("x-request-id") ?? null;
}

/**
 * Check for cached response. Returns entry if valid,
 * null and cleans up if expired.
 */
export function checkIdempotency(key: string) {
  const entry = CACHE.get(key);
  if (entry && entry.expires > Date.now()) {
    return entry;
  }
  // Cleanup expired entry
  if (entry) CACHE.delete(key);
  return null;
}

/**
 * Store response for future deduplication.
 */
export function saveIdempotency(
  key: string,
  body: string,
  status: number,
  windowMs: number = DEFAULT_WINDOW_MS
): void {
  CACHE.set(key, {
    status,
    body,
    expires: Date.now() + windowMs,
  });
}

Handler Integration: open-sse/handlers/chatCore/idempotency.ts

import {
  getIdempotencyKey,
  checkIdempotency,
  saveIdempotency,
} from "@/lib/idempotencyLayer";

export async function handleIdempotentRequest(
  request: Request,
  execute: () => Promise<Response>
): Promise<Response> {
  // Step 1: Extract possible idempotency key
  const key = getIdempotencyKey(request.headers);

  // Step 2: Check cache if key exists
  if (key) {
    const cached = checkIdempotency(key);
    if (cached) {
      // Return identical response without re-execution
      return new Response(cached.body, {
        status: cached.status,
        headers: { "X-Idempotency-Replay": "true" },
      });
    }
  }

  // Step 3: Execute normal processing
  const response = await execute();
  const body = await response.clone().text();

  // Step 4: Cache result for retries
  if (key) {
    saveIdempotency(key, body, response.status);
  }

  return response;
}

Configuration and Operational Characteristics

Aspect	Default	Override
Cache window	5 minutes (300,000 ms)	OMNIROUTE_IDEMPOTENCY_WINDOW_MS env var
Storage	In-memory Map	N/A (process-local)
Key headers	idempotency-key, x-request-id	None
Cleanup	On-access expiration check	Automatic on lookup

Single-Process Design Trade-offs

The OmniRoute idempotency cache is process-local by design. Each running instance maintains its own isolated Map. This approach provides:

• Zero network latency for cache operations
• No external dependency for standalone deployments
• Simplicity suitable for single-instance serverless or containerized runs

For multi-instance deployments behind a load balancer, requests with the same idempotency key must route to the same instance, or an external store (Redis, DynamoDB) must replace the in-memory Map.

Summary

• getIdempotencyKey() extracts identifiers from idempotency-key or x-request-id headers in src/lib/idempotencyLayer.ts
• checkIdempotency() returns cached responses immediately, skipping expensive re-execution
• saveIdempotency() stores successful responses with 5-minute TTL (configurable)
• Integration in open-sse/handlers/chatCore/idempotency.ts wraps the full chat request pipeline
• In-memory Map with expiration provides sub-millisecond lookup without external dependencies
• Environment-configurable window adapts to different retry patterns and compliance requirements

Frequently Asked Questions

What happens if no idempotency key is provided?

Requests without an idempotency-key or x-request-id header bypass the cache entirely. The getIdempotencyKey() function returns null, and the handler executes the full request flow without caching. This ensures backward compatibility and avoids cache pollution from non-idempotent operations.

How does OmniRoute handle cache expiration?

Expiration is lazy: the checkIdempotency() function validates the expires timestamp on every lookup. Stale entries are deleted immediately and return null, triggering normal execution. There is no background cleanup process, keeping the implementation simple and deterministic.

Can I use Redis or another external store instead?

The current implementation in src/lib/idempotencyLayer.ts uses a plain JavaScript Map. For distributed deployments, you would replace the CACHE variable and the three exported functions with adapters to your external store. The interface remains identical, so the chat handler requires no changes.

What response status codes are cached?

OmniRoute caches all responses regardless of status code, including 4xx and 5xx errors. This prevents thundering-herd problems where a failing downstream service receives duplicate failing requests. Clients receive consistent results for their retries within the window.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}