How to Implement Incremental Builds in Gatsby: CI/CD Optimization Guide

Gatsby incremental builds regenerate only pages affected by data changes by persisting the .cache and public directories between runs, cutting build times from minutes to seconds when properly configured in CI/CD pipelines.

The gatsbyjs/gatsby repository implements incremental builds as a core runtime feature since version 3.0, automatically tracking node changes and selectively rendering HTML without requiring explicit flags in open-source deployments. This guide explains the underlying mechanisms, critical source files, and CI/CD configurations needed to maintain persistent state across builds.

How Incremental Builds Work in Gatsby

Gatsby’s incremental build system operates through a sophisticated dirty-page tracking mechanism that compares current data against cached state. The implementation relies on several interconnected subsystems within the core packages.

Cache Preservation and State Tracking

The foundation of incremental builds rests on two directories: .cache and public. During the build process, Gatsby stores the graph structure, schema metadata, and node data in .cache, while generated HTML assets reside in public.

According to the source code in packages/gatsby/src/schema/index.js, the build system initializes "build metadata for type inference and start[s] updating it incrementally." Underlying this persistence is the LMDB datastore implementation found in packages/gatsby/src/datastore/lmdb/lmdb-datastore.ts, which provides the transactional storage layer for node data between builds.

Node Change Detection

During the data-sourcing phase, source plugins (such as gatsby-source-wordpress or gatsby-source-contentful) mark nodes as added, updated, or deleted. The runtime maintains this state in the Redux store, specifically within packages/gatsby/src/redux/reducers/inference-metadata.ts, which contains the incrementalBuild step flag and tracks which pages depend on modified nodes.

When a node changes, the system flags all connected pages as "dirty" using internal utilities like findConnectedNodes, which invokes touchNode to propagate change notifications through the dependency graph.

Selective Page Regeneration

When the HTML generation step executes, Gatsby checks the internal dirtyPages set. Only pages appearing in this set are passed to the SSR pipeline for regeneration; unchanged pages are copied directly from the previous public output. This selective approach bypasses the costly process of rebuilding unchanged static assets.

The step: StepsEnum.incrementalBuild logic in the inference metadata reducer controls this gating mechanism, ensuring that builds scale with the magnitude of content changes rather than total site size.

Requirements for CI/CD Optimization

To achieve sub-minute builds in continuous integration environments, you must configure artifact persistence explicitly. Without proper caching, each CI run performs a full rebuild, negating the benefits of incremental compilation.

Persisting Cache Between Runs

Your CI pipeline must preserve both .cache and public as build artifacts. The general pattern involves:

1. Restoring cached directories at the start of each job
2. Executing gatsby build (or npm run build)
3. Saving the updated directories as artifacts for subsequent runs

Most platforms (GitHub Actions, CircleCI, Netlify) provide caching mechanisms that support this workflow. The cache key should incorporate the runner OS and commit hash, with fallback restore keys to maximize hit rates.

Configuring Build Commands

Avoid executing gatsby clean in CI environments unless specifically requiring a full rebuild. The clean command deletes .cache and public, forcing the next build to start from scratch. Instead, rely on Gatsby’s internal dirty-checking logic to determine which pages require regeneration.

In Gatsby v3 and later, incremental builds are enabled by default with no environment flags required. For legacy Gatsby v2 deployments, you must set GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=1.

Unsupported Patterns to Avoid

Certain coding patterns disable incremental builds automatically. Direct usage of Node.js fs modules within gatsby-ssr.js or other SSR files triggers the unsafe filesystem detection, forcing a full rebuild. To maintain incremental capabilities, use ES-module imports or Gatsby’s data layer APIs exclusively, avoiding side effects that bypass the tracked node system.

Implementation Examples

Basic Gatsby Configuration

No special configuration is required for standard incremental builds in Gatsby v3+. A typical gatsby-config.js with source plugins automatically participates in the incremental pipeline:

module.exports = {
  plugins: [
    {
      resolve: `gatsby-source-wordpress`,
      options: {
        url: `https://example.com/graphql`,
      },
    },
    `gatsby-plugin-image`,
  ],
}

GitHub Actions Workflow

The following configuration persists cache directories between runs using actions/cache:

name: Gatsby CI

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Cache Gatsby directories
        uses: actions/cache@v3
        with:
          path: |
            .cache
            public
          key: ${{ runner.os }}-gatsby-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-gatsby-

      - name: Install dependencies
        run: npm ci

      - name: Build
        run: npm run build

Disabling Incremental Builds (Debug Mode)

In rare cases requiring full rebuilds for debugging, you can explicitly disable the feature:

// gatsby-node.js
exports.onCreateWebpackConfig = ({ actions }) => {
  actions.setWebpackConfig({
    plugins: [
      new webpack.DefinePlugin({
        GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES: JSON.stringify(false)
      })
    ],
  })
}

Summary

• Gatsby incremental builds rely on persistent .cache and public directories to track state between runs.
• The system uses LMDB datastores and Redux reducers (specifically inference-metadata.ts) to track dirty pages and node dependencies.
• CI/CD optimization requires configuring artifact caching to preserve these directories; avoid gatsby clean in automated pipelines.
• Gatsby v3+ enables incremental builds by default, while v2 requires experimental flags.
• Unsafe filesystem access in SSR code disables incremental builds automatically.

Frequently Asked Questions

Do I need Gatsby Cloud to use incremental builds?

No. Incremental builds are available in the open-source Gatsby core since version 3.0. While Gatsby Cloud provides additional optimizations like Deferred Static Generation (DSG) and Parallel Rendering, the fundamental incremental build capability works in any environment where the .cache and public directories persist between builds.

What happens if I run gatsby clean in CI?

Running gatsby clean deletes the .cache and public directories, forcing the subsequent build to perform a full regeneration of all pages. This eliminates the performance benefits of incremental builds. Only use this command when explicitly requiring a fresh build due to schema changes or corrupted cache states.

Which Gatsby version supports incremental builds?

Gatsby v3.0 and later support incremental builds by default without configuration. Gatsby v2 supported incremental builds experimentally using the GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES environment variable, though this is deprecated in favor of the automatic implementation in v3+.

How do source plugins support incremental builds?

Source plugins participate in the incremental pipeline by calling touchNode for modified content, which flags dependent pages as dirty. The internal findConnectedNodes utility (implemented in plugins like gatsby-source-wordpress) identifies which pages query specific nodes, ensuring only affected pages enter the dirtyPages set for regeneration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}