How the Gatsby Cache System Works: Build Performance Optimization Guide

Gatsby's cache system uses a layered multi-cache architecture combining an in-memory LRU store with a persistent disk cache to minimize redundant work during builds, and you can significantly improve performance by switching to the LMDB-backed implementation and tuning memory limits.

The Gatsby cache system is a critical component of the gatsbyjs/gatsby repository that persists intermediate build data between runs to avoid redundant processing. By storing query results, page data, and transformed assets in a layered cache, Gatsby dramatically reduces build times for subsequent runs. Understanding how this system works under the hood allows you to optimize your own plugins and build pipelines for maximum performance.

Core Architecture of the Gatsby Cache System

The cache implementation lives primarily in packages/gatsby/src/utils/cache.ts, where the GatsbyCache class orchestrates a multi-layer storage strategy.

Multi-Layer Caching Strategy

Each GatsbyCache instance composes two distinct stores using the cache-manager library:

1. Memory cache – An LRU-style store limited to MAX_CACHE_SIZE (250 entries by default) with a practically infinite TTL. This provides microsecond-level access for hot data.
2. Disk cache – A persistent store written to .cache/caches/<name> on the filesystem, ensuring data survives process restarts.

The init() method in cache.ts wires these together into a multiCaching instance:

// packages/gatsby/src/utils/cache.ts
export default class GatsbyCache {
  // ...
  init(): GatsbyCache {
    fs.ensureDirSync(this.directory)

    const configs: Array<StoreConfig> = [
      { store: `memory`, max: MAX_CACHE_SIZE, ttl: TTL },          // memory layer
      { store: this.store, ttl: TTL, options: { path: this.directory } }, // disk layer
    ]

    const caches = configs.map(cache => manager.caching(cache))
    this.cache = manager.multiCaching(caches)                      // combines both
    return this
  }
}

Disk Store Implementation

The filesystem persistence layer is implemented in packages/gatsby/src/cache/cache-fs.ts. The DiskStore class handles low-level read/write logic with several optimizations:

• Key hashing – Cache keys are hashed using md5 to generate deterministic filenames (diskstore-<hash>), preventing filesystem issues with special characters.
• Sub-directory sharding – When the subdirs option is enabled, files are distributed across subdirectories (using the first 3 characters of the hash) to avoid directory size limits.
• File locking – A globalGatsbyCacheLock ensures safe concurrent reads/writes during a single build process.

// packages/gatsby/src/cache/cache-fs.ts (excerpt)
DiskStore.prototype._getFilePathByKey = function (key): string {
  const hash = crypto.createHash(`md5`).update(key + ``).digest(`hex`)
  return this.options.subdirs
    ? path.join(this.options.path, `diskstore-` + hash.slice(0, 3), hash.slice(3))
    : path.join(this.options.path, `diskstore-` + hash)
}

The actual JSON serialization (including optional gzip compression for large buffers) is handled by packages/gatsby/src/cache/json-file-store.ts, which externalizes large Buffer objects to separate .bin files when necessary.

LMDB Cache Alternative

Since Gatsby v4, the framework offers an LMDB-backed cache (GatsbyCacheLmdb) implemented in packages/gatsby/src/utils/cache-lmdb.ts. LMDB (Lightning Memory-Mapped Database) stores data in a memory-mapped B-tree, providing significantly lower latency than the JSON file-system store, especially for thousands of small entries like query hash lookups.

The LMDB implementation uses the same API (get, set, del) as the standard cache, making substitution seamless:

// packages/gatsby/src/utils/cache-lmdb.ts (excerpt)
private static getStore(): RootDatabase {
  // ...
  rootDb = open({ name: `root`, path: dbPath, compression: true, maxDbs: 200 })
  // ...
}

Where Gatsby Uses Caching Internally

Gatsby leverages its cache system across several critical build paths to avoid redundant computation:

• Query Runner – resultHashCache in packages/gatsby/src/query/query-runner.ts stores hashed query results to skip re-running identical queries.
• Page Data Generation – lmdbPageQueryResultsCache in packages/gatsby/src/utils/page-data.ts caches serialized page data between builds.
• Date Formatting – formatDateCache in packages/gatsby/src/schema/types/date.ts memoizes expensive date formatting operations.

These internal caches demonstrate how persisting intermediate state dramatically reduces build times for large sites.

Strategies to Improve Gatsby Build Performance

Optimizing the Gatsby cache system involves tuning both the memory and persistence layers. Here are the most effective strategies:

Strategy	What It Does	How to Apply
Enable LMDB cache	Replaces slow JSON-file disk cache with a memory-mapped database.	Set environment variable GATSBY_CACHE_LMDB=true or instantiate new GatsbyCacheLmdb({ name: 'cacheName' }).init() in custom plugins.
Increase memory cache size	Retains more hot data in RAM, avoiding disk I/O.	Adjust MAX_CACHE_SIZE in packages/gatsby/src/utils/cache.ts or set process.env.GATSBY_MAX_CACHE_SIZE to a higher value than the default 250.
Enable compression	Reduces disk I/O for large JSON payloads.	Pass { zip: true } to the DiskStore options when constructing GatsbyCache, or ensure compression: true is set in LMDB initialization.
Use sub-directories	Prevents filesystem slowdown when a cache grows beyond thousands of files.	Set subdirs: true in DiskStore options to shard files across subdirectories using hash prefixes.
Clear stale caches	Removes orphaned files that waste space and cause lock contention.	Run await cache.reset() on GatsbyCache or GatsbyCacheLmdb instances, or delete .cache/caches between major version upgrades.
Leverage incremental builds	Re-uses previously persisted cache across builds instead of rebuilding from scratch.	Ensure the .cache folder persists between CI runs by mounting it as a cache artifact; avoid clearing it between gatsby develop sessions.
Avoid duplicate cache instances	Prevents redundant I/O when multiple plugins use the same cache name.	Share a single cache instance via require("gatsby/src/utils/get-cache") or export a common cache from a utility module.

Practical Code Examples

To implement these optimizations in your own plugins or site configuration, use the following patterns:

Using LMDB for Custom Plugin Caching

// src/custom-plugin.ts – using the LMDB cache for expensive image processing
import GatsbyCacheLmdb from "gatsby/src/utils/cache-lmdb"

const imageCache = new GatsbyCacheLmdb({ name: `image-processing` }).init()

export async function processImage(hash: string, src: string) {
  // 1️⃣ Try to read from cache
  const cached = await imageCache.get<string>(hash)
  if (cached) return cached

  // 2️⃣ Perform heavy work (e.g., sharp transformations)
  const result = await heavyTransform(src)

  // 3️⃣ Store result for next run
  await imageCache.set(hash, result)

  return result
}

Resetting Caches After Major Updates

// src/utility.ts – resetting caches after a major version bump
import { resetCache } from "gatsby/src/utils/cache-lmdb"

export async function cleanAllCaches() {
  // Clears LMDB caches
  await resetCache()
  // Clears FS caches (if you also use GatsbyCache)
  // await new GatsbyCache().init().reset()
}

Key Source Files

Understanding the Gatsby cache system requires familiarity with these specific modules:

File	Role	Link
packages/gatsby/src/utils/cache.ts	High-level multi-cache factory (memory + disk)	cache.ts
packages/gatsby/src/cache/cache-fs.ts	Low-level file-system store, locking, key hashing	cache-fs.ts
packages/gatsby/src/cache/json-file-store.ts	JSON (and optional gzip) serialization, external buffer handling	json-file-store.ts
packages/gatsby/src/utils/cache-lmdb.ts	LMDB-backed cache implementation (fast, memory-mapped)	cache-lmdb.ts
packages/gatsby/src/query/query-runner.ts	Example consumer: caches query hash results	query-runner.ts
packages/gatsby/src/utils/page-data.ts	Example consumer: caches page query results	page-data.ts

Summary

• The Gatsby cache system implements a multi-layer architecture in packages/gatsby/src/utils/cache.ts, combining a fast in-memory LRU cache (250 entries by default) with a persistent disk cache stored in .cache/caches/<name>.
• Disk persistence uses hashed filenames and optional sub-directory sharding via packages/gatsby/src/cache/cache-fs.ts, with file locking to prevent race conditions during builds.
• LMDB integration (available since Gatsby v4) in packages/gatsby/src/utils/cache-lmdb.ts offers a high-performance alternative to JSON file storage, using memory-mapped B-trees for lower latency.
• To optimize build performance, enable LMDB caching, increase memory cache limits, enable compression, and persist the .cache directory between CI runs to leverage incremental builds.

Frequently Asked Questions

How do I clear the Gatsby cache when builds behave unexpectedly?

Run gatsby clean from your terminal to delete the .cache and public directories entirely. For programmatic control in custom plugins, invoke await cache.reset() on your GatsbyCache or GatsbyCacheLmdb instance to clear specific cache namespaces without wiping the entire build directory.

Should I use the LMDB cache or the standard file-system cache for my plugin?

Choose LMDB (GatsbyCacheLmdb) when your plugin performs frequent small reads/writes (such as query result caching or date formatting), as the memory-mapped B-tree structure in packages/gatsby/src/utils/cache-lmdb.ts provides lower latency than JSON file I/O. Use the standard GatsbyCache from packages/gatsby/src/utils/cache.ts for larger, infrequently accessed payloads where compression and simple file storage suffice.

How does Gatsby handle cache persistence between CI builds?

Gatsby writes all cache data to the .cache directory at the project root, including both the multi-layer disk stores and LMDB databases. To enable incremental builds in CI environments, configure your build platform (Netlify, Vercel, GitHub Actions, etc.) to preserve the .cache folder as a build artifact between runs, ensuring that gatsby build can reuse previously computed query results and page data.

What is the default memory cache limit and how do I increase it?

The default MAX_CACHE_SIZE is 250 entries as defined in packages/gatsby/src/utils/cache.ts. To increase this limit and keep more hot data in RAM, set the environment variable GATSBY_MAX_CACHE_SIZE to a higher value (e.g., 500 or 1000) before running your build, or modify the MAX_CACHE_SIZE constant directly in a forked version of the cache utility if you require persistent changes across projects.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}