How the 3-Tier Caching System in WorldMonitor Prevents Cache Stampedes

The 3-tier caching system prevents cache stampedes by combining process-local in-memory maps, Redis-based request coalescing, and negative caching to ensure only one upstream request executes per key even during traffic spikes.

WorldMonitor, an open-source API aggregation platform (koala73/worldmonitor), serves millions of requests through edge functions. To protect upstream data providers from overload during traffic surges, the project implements a sophisticated 3-tier caching system that eliminates cache stampedes through progressive fallback layers and request deduplication.

Understanding Cache Stampedes and the 3-Tier Architecture

A cache stampede (or thundering herd) occurs when a cached item expires and multiple simultaneous requests attempt to regenerate it, overwhelming the upstream database or API. WorldMonitor's architecture defends against this through three distinct layers:

1. In-memory process cache – Local Map objects for hot-path deduplication
2. Redis with request coalescing – Shared cache using single-flight pattern via inflight tracking
3. Upstream with negative caching – Sentinel values preventing repeated fetches for missing data

Layer 1: Process-Local In-Memory Cache

The first line of defense lives directly within the Edge function process. In server/worldmonitor/market/v1/list-commodity-quotes.ts, a module-level Map stores recent results:

const fallbackCommodityCache = new Map<
  string,
  { data: ListCommodityQuotesResponse; ts: number }
>();

After retrieving fresh data from Redis, the handler stores the result in this local map:

// After successful Redis fetch
fallbackCommodityCache.set(redisKey, { data: result, ts: Date.now() });

Subsequent requests within the same process check this map first:

const cached = fallbackCommodityCache.get(redisKey);
if (cached && Date.now() - cached.ts < TTL_MS) {
  return cached.data;
}

Why this prevents stampedes: Even if thousands of requests hit the same edge node, only the first triggers a Redis lookup. The rest are served from local memory, eliminating network traffic and reducing Redis load to a single request per key per instance.

Layer 2: Redis with Request Coalescing

When the in-memory cache misses, the system falls back to Redis. The critical protection mechanism resides in server/_shared/redis.ts within the cachedFetchJson function, which implements a single-flight pattern using an inflight map:

const inflight = new Map<string, Promise<unknown>>();

export async function cachedFetchJson<T extends object>(
  key: string,
  ttlSeconds: number,
  fetcher: () => Promise<T | null>,
  negativeTtlSeconds = 120
): Promise<T | null> {
  // Check Redis first
  const cached = await getCachedJson(key);
  if (cached !== null) return cached as T;

  // Critical: Check if someone else is already fetching this key
  const existing = inflight.get(key);
  if (existing) return existing as Promise<T | null>;

  // First caller becomes the leader
  const promise = fetcher()
    .then(async (result) => {
      if (result != null) {
        await setCachedJson(key, result, ttlSeconds);
      } else {
        await setCachedJson(key, NEG_SENTINEL, negativeTtlSeconds);
      }
      return result;
    })
    .finally(() => inflight.delete(key));

  inflight.set(key, promise);
  return promise;
}

How coalescing works: When multiple concurrent requests miss the Redis cache for the same key, the inflight map ensures only the first request executes the fetcher() function. All other requests receive the same Promise, awaiting the result without triggering additional upstream calls.

Negative Caching for Missing Data

The system prevents stampedes for non-existent resources through negative caching. When the upstream returns null, cachedFetchJson stores a sentinel value (__WM_NEG__) with a short TTL (default 120 seconds):

const NEG_SENTINEL = '__WM_NEG__';

// Inside the promise resolution:
if (result != null) {
  await setCachedJson(key, result, ttlSeconds);
} else {
  await setCachedJson(key, NEG_SENTINEL, negativeTtlSeconds);
}

This ensures that repeated requests for missing data hit the cached sentinel rather than bombarding the upstream with futile requests.

Layer 3: Upstream Protection

The upstream data providers (Yahoo Finance, GDELT, etc.) are shielded by the preceding layers. By the time a request reaches the upstream fetcher() function, the system has already:

1. Checked the local in-memory cache (Layer 0)
2. Checked Redis (Layer 1)
3. Ensured only one request per key executes via inflight coalescing (Layer 1)
4. Cached negative results to prevent retry loops (Layer 1)

This architecture guarantees that traffic spikes result in at most one upstream request per cache key, regardless of how many edge instances or concurrent connections are active.

End-to-End Flow Example

Consider a sudden surge of 10,000 requests for commodity quotes for the symbol "GOLD":

1. In-memory check: Each edge instance (say 50 instances) checks local fallbackCommodityCache. All miss initially.
2. Redis check: Each instance queries Redis. All miss (key not yet set).
3. In-flight coalescing:
   • First request across all instances reaches cachedFetchJson and becomes the "leader"
   • The other 9,999 requests find the key in inflight and await the same promise
4. Upstream fetch: Single request to Yahoo Finance API
5. Cache population: Result stored in Redis (10 min TTL) and returned to all 10,000 waiting promises
6. In-memory update: Each instance stores result in fallbackCommodityCache
7. Subsequent requests: New requests hit local memory instantly, zero network calls

Summary

• Process-local Maps in list-commodity-quotes.ts eliminate redundant Redis calls within the same edge instance through hot-path deduplication.
• Request coalescing via the inflight Map in cachedFetchJson ensures only one upstream fetch executes per key across all instances, with concurrent requests sharing the same promise.
• Negative caching using the __WM_NEG__ sentinel prevents thundering herds for missing data by caching "not found" responses with a short TTL.
• Progressive fallback from in-memory → Redis → upstream guarantees that traffic spikes never overwhelm external data providers.

Frequently Asked Questions

What is a cache stampede?

A cache stampede occurs when a cached item expires or a cache miss happens, causing numerous simultaneous requests to hit the backend database or API simultaneously to regenerate the same data. This sudden surge can overwhelm upstream services, causing cascading failures and degraded performance across the system.

How does request coalescing work in WorldMonitor?

WorldMonitor implements request coalescing through an inflight Map in server/_shared/redis.ts. When multiple requests miss the Redis cache for the same key simultaneously, only the first request executes the upstream fetcher function. This "leader" request stores its promise in the inflight map, and all other concurrent requests receive the same promise object, effectively waiting for the single upstream call to complete without triggering additional fetches.

What is negative caching and why is it used?

Negative caching is the practice of storing a sentinel value (in WorldMonitor, __WM_NEG__) when upstream data is missing or returns null, rather than leaving the cache empty. This prevents a "thundering herd" of requests for non-existent resources, as subsequent requests find the cached sentinel value and return immediately instead of repeatedly querying the upstream service. WorldMonitor applies a shorter TTL (default 120 seconds) to negative entries to allow for eventual data availability while still protecting the upstream.

Can this 3-tier pattern be used in other Node.js applications?

Yes, the 3-tier caching pattern implemented in WorldMonitor is highly portable to other Node.js applications facing similar stampede risks. The key components—process-local Maps for hot-path optimization, Redis with single-flight request coalescing via an inflight map, and negative caching with sentinel values—can be adapted to any Node.js service using libraries like ioredis or redis. The pattern is particularly effective in serverless or edge environments where multiple instances may cold-start simultaneously and bombard shared backends.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}