# How the Idempotency Cache Works in OmniRoute: In-Memory Deduplication for Reliable API Requests

> Learn how OmniRoute's idempotency cache uses an in-memory Map to deduplicate API requests by Idempotency-Key header, ensuring reliable retries with cached responses.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-09-01

---

**OmniRoute uses a lightweight in-memory Map-based cache to deduplicate requests by their `Idempotency-Key` header, returning cached responses for retries within a configurable time window.**

The **idempotency cache in OmniRoute** protects downstream providers from redundant calls and ensures exactly-once semantics for safe retries. This mechanism is essential for handling network timeouts, client retries, and duplicate submissions without re-executing expensive operations like LLM inference or quota-tracked requests.

## Core Architecture of the Idempotency Cache

The cache implementation spans two primary layers: a reusable library module and a handler-specific integration point. Together they provide a deterministic key extraction, lookup, and storage pipeline.

### Key Extraction: `getIdempotencyKey()` in [`src/lib/idempotencyLayer.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/idempotencyLayer.ts)

The entry point for all idempotent requests is the `getIdempotencyKey()` function. This utility extracts the idempotency identifier from incoming headers, supporting both standard `Headers` objects and plain object maps for flexibility across different request types.

The function checks for two header variants in priority order:

1. **`idempotency-key`** — the standard header per [RFC 7239](https://tools.ietf.org/html/rfc7239) conventions
2. **`x-request-id`** — legacy fallback for backward compatibility

If neither header is present, the function returns `null` and idempotency is bypassed for that request.

### Cache Storage: In-Memory Map with TTL

The underlying data structure is a simple `Map<string, IdempotentEntry>` stored in process memory. Each entry contains:

| Property | Type | Purpose |
|----------|------|---------|
| `status` | `number` | HTTP status code to return |
| `body` | `string` | Serialized response body |
| `expires` | `number` | Timestamp when the entry becomes invalid |

The default expiration window is **`DEFAULT_WINDOW_MS`** (5 minutes), configurable via the `OMNIROUTE_IDEMPOTENCY_WINDOW_MS` environment variable.

### Lookup: `checkIdempotency()` Returns Cached Responses

The `checkIdempotency(key)` function performs a fast Map lookup with automatic expiration handling:

- **Hit within window**: Returns the full entry immediately
- **Expired entry**: Deletes the stale entry and returns `null`
- **Miss**: Returns `null`, allowing normal request processing

### Storage: `saveIdempotency()` Captures Outgoing Responses

After successful request execution, the `saveIdempotency()` function persists the response. This ensures that any immediate retry—common when clients experience timeouts—hits the cache rather than re-triggering the operation.

## Integration in Chat Core Handler: [`open-sse/handlers/chatCore/idempotency.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore/idempotency.ts)

The chat processing pipeline wraps its execution logic with the idempotency layer. According to the OmniRoute source code, this handler orchestrates the full lifecycle:

1. Extract key from request headers
2. Check cache for existing response
3. Return cached result immediately on hit
4. Execute normal flow on miss
5. Store final response for future retries

This integration point is critical because chat requests often involve:
- **Expensive LLM provider calls** with per-token pricing
- **Quota tracking** that must not increment on duplicates
- **Streaming responses** that get buffered and cached as complete payloads

## Complete Implementation Example

### Cache Library: [`src/lib/idempotencyLayer.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/idempotencyLayer.ts)

```typescript
// Core cache storage with TTL-based expiration
const CACHE = new Map<string, {
  status: number;
  body: string;
  expires: number;
}>();

const DEFAULT_WINDOW_MS = 5 * 60 * 1000; // 5 minutes

/**
 * Extract idempotency key from various header formats.
 * Supports both Headers API and plain objects.
 */
export function getIdempotencyKey(
  headers: Headers | Record<string, string>
): string | null {
  const get =
    typeof (headers as Headers).get === "function"
      ? (k: string) => (headers as Headers).get(k)
      : (k: string) => (headers as Record<string, string>)[k];

  return get("idempotency-key") ?? get("x-request-id") ?? null;
}

/**
 * Check for cached response. Returns entry if valid,
 * null and cleans up if expired.
 */
export function checkIdempotency(key: string) {
  const entry = CACHE.get(key);
  if (entry && entry.expires > Date.now()) {
    return entry;
  }
  // Cleanup expired entry
  if (entry) CACHE.delete(key);
  return null;
}

/**
 * Store response for future deduplication.
 */
export function saveIdempotency(
  key: string,
  body: string,
  status: number,
  windowMs: number = DEFAULT_WINDOW_MS
): void {
  CACHE.set(key, {
    status,
    body,
    expires: Date.now() + windowMs,
  });
}

```

### Handler Integration: [`open-sse/handlers/chatCore/idempotency.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore/idempotency.ts)

```typescript
import {
  getIdempotencyKey,
  checkIdempotency,
  saveIdempotency,
} from "@/lib/idempotencyLayer";

export async function handleIdempotentRequest(
  request: Request,
  execute: () => Promise<Response>
): Promise<Response> {
  // Step 1: Extract possible idempotency key
  const key = getIdempotencyKey(request.headers);

  // Step 2: Check cache if key exists
  if (key) {
    const cached = checkIdempotency(key);
    if (cached) {
      // Return identical response without re-execution
      return new Response(cached.body, {
        status: cached.status,
        headers: { "X-Idempotency-Replay": "true" },
      });
    }
  }

  // Step 3: Execute normal processing
  const response = await execute();
  const body = await response.clone().text();

  // Step 4: Cache result for retries
  if (key) {
    saveIdempotency(key, body, response.status);
  }

  return response;
}

```

## Configuration and Operational Characteristics

| Aspect | Default | Override |
|--------|---------|----------|
| Cache window | 5 minutes (300,000 ms) | `OMNIROUTE_IDEMPOTENCY_WINDOW_MS` env var |
| Storage | In-memory `Map` | N/A (process-local) |
| Key headers | `idempotency-key`, `x-request-id` | None |
| Cleanup | On-access expiration check | Automatic on lookup |

### Single-Process Design Trade-offs

The OmniRoute idempotency cache is **process-local by design**. Each running instance maintains its own isolated Map. This approach provides:

- **Zero network latency** for cache operations
- **No external dependency** for standalone deployments
- **Simplicity** suitable for single-instance serverless or containerized runs

For multi-instance deployments behind a load balancer, requests with the same idempotency key must route to the same instance, or an external store (Redis, DynamoDB) must replace the in-memory Map.

## Summary

- **`getIdempotencyKey()`** extracts identifiers from `idempotency-key` or `x-request-id` headers in [`src/lib/idempotencyLayer.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/idempotencyLayer.ts)
- **`checkIdempotency()`** returns cached responses immediately, skipping expensive re-execution
- **`saveIdempotency()`** stores successful responses with 5-minute TTL (configurable)
- **Integration in [`open-sse/handlers/chatCore/idempotency.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore/idempotency.ts)** wraps the full chat request pipeline
- **In-memory Map with expiration** provides sub-millisecond lookup without external dependencies
- **Environment-configurable window** adapts to different retry patterns and compliance requirements

## Frequently Asked Questions

### What happens if no idempotency key is provided?

Requests without an `idempotency-key` or `x-request-id` header bypass the cache entirely. The `getIdempotencyKey()` function returns `null`, and the handler executes the full request flow without caching. This ensures backward compatibility and avoids cache pollution from non-idempotent operations.

### How does OmniRoute handle cache expiration?

Expiration is **lazy**: the `checkIdempotency()` function validates the `expires` timestamp on every lookup. Stale entries are deleted immediately and return `null`, triggering normal execution. There is no background cleanup process, keeping the implementation simple and deterministic.

### Can I use Redis or another external store instead?

The current implementation in [`src/lib/idempotencyLayer.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/idempotencyLayer.ts) uses a plain JavaScript `Map`. For distributed deployments, you would replace the `CACHE` variable and the three exported functions with adapters to your external store. The interface remains identical, so the chat handler requires no changes.

### What response status codes are cached?

OmniRoute caches **all responses** regardless of status code, including 4xx and 5xx errors. This prevents thundering-herd problems where a failing downstream service receives duplicate failing requests. Clients receive consistent results for their retries within the window.