How the AI Engineering from Scratch Website Is Generated: Node.js Build Pipeline Explained

The AI Engineering from Scratch website is built automatically by a Node.js script (site/build.js) that parses Markdown curriculum files and compiles them into static HTML, JavaScript data bundles, and SEO manifests.

The rohitg00/ai-engineering-from-scratch repository uses a custom static site generator to transform its curriculum Markdown into a fully functional learning platform. Unlike traditional static site generators like Jekyll or Hugo, this project employs a bespoke Node.js build pipeline that processes README.md, ROADMAP.md, and lesson documentation to produce the AI Engineering from Scratch website. The result is a completely static site served directly from the site/ folder without any server-side rendering requirements.

The Build Pipeline Architecture

According to the source code in site/build.js, the generation process follows a deterministic nine-step pipeline. When a commit lands on the main branch, a GitHub Action defined in .github/workflows/curriculum.yml executes node site/build.js, triggering the following automated workflow.

Step 1: Ingesting Core Documentation

The build script begins by loading the canonical curriculum sources from the repository root. As implemented in lines 4‑6 of site/build.js, the script reads:

• README.md — Contains the master lesson table with phases, URLs, and metadata
• ROADMAP.md — Tracks implementation status using emoji indicators (✅/🚧/⬚)
• glossary/terms.md — Provides terminology definitions

Step 2: Parsing Curriculum Structure

Two primary functions extract structured data from the raw Markdown:

parseRoadmap() (lines 4‑34) scans ROADMAP.md to extract the completion status of every phase and lesson, mapping emoji indicators to Boolean or tri-state values.

parseReadme() (lines 38‑88) walks the Markdown tables in README.md to build a canonical registry of phases and lessons. This includes extracting lesson names, URLs, content types, programming languages, and cross-referencing them against the roadmap status.

Step 3: Discovering Learning Path Overlays

The parseLearningPaths() function (lines 7‑45) scans the learning-paths/ directory for JSON configuration files. These overlays create ordered learning tracks that reference the canonical lessons parsed from README.md, allowing the site to present alternative navigation sequences without duplicating content.

Step 4: Resolving Interactive Figure Dependencies

The build system maintains a sophisticated figure provider architecture for interactive diagrams:

1. discoverUsedFigureIds() (lines 35‑45) recursively scans all *.md files in phases/ and certifications/ directories, searching for ```figure code fences and extracting unique figure IDs.
2. buildFigureProviderManifest() (lines 58‑90) matches discovered figure IDs against the ordered list of provider scripts defined in FIGURE_PROVIDER_ORDER.
3. The resulting manifest is serialized to site/figure-manifest.js, establishing the mapping between figure IDs and their rendering implementations.

Step 5: Generating SEO Metadata

The buildSeoManifests() function (lines 19‑28) processes each lesson's docs/en.md file to extract:

• Title and summary descriptions
• Keywords and excerpts
• Canonical URLs

It outputs two JSON files:

• site/lesson-seo.json — Metadata for individual lessons
• site/certification-seo.json — Metadata for certification tracks

Step 6: Rendering Discovery Interfaces

Using the SEO data structures, the script generates HTML discovery tables:

• renderCatalogDiscovery() (lines 38‑55) produces HTML rows for the main curriculum catalog.
• renderCertificationDiscovery() generates rows for the certifications page.

These HTML fragments are inserted into site/catalog.html and site/certifications.html between special marker comments, replacing previous generated content.

Step 7: Writing Output Artifacts

The final phase writes all generated assets to disk (lines 24‑33 and 81‑92):

• site/data.js — A JavaScript module exposing window.AIFS_DATA, containing the complete phases/lessons hierarchy, metadata, and figure mappings.
• site/figure-manifest.js — The figure provider registry.
• site/lesson-seo.json and site/certification-seo.json — SEO manifests.
• Updated catalog.html and certifications.html with fresh discovery table markup.

Continuous Integration and Deployment

The repository uses GitHub Actions for automated regeneration. The workflow curriculum.yml triggers on every push to main, executes node site/build.js, then commits the regenerated site/data.js (the only generated file versioned in the repository) back to the main branch. This ensures the AI Engineering from Scratch website always reflects the latest curriculum changes without manual intervention.

Working with the Generated Data

Because the site is purely static, all dynamic behavior relies on the JavaScript data bundles created by the build process.

Accessing Curriculum Data Client-Side

After the build completes, site/data.js creates a global data structure:

// Loaded via <script src="data.js"></script>
const phases = window.AIFS_DATA.phases;
const totalLessons = phases.reduce((sum, p) => sum + p.lessons.length, 0);

// Access a specific lesson
const phase0 = phases.find(p => p.id === 0);
const lesson = phase0.lessons.find(l => l.id === 'setup-environment');
console.log(lesson.name, lesson.url);

Resolving Figure Providers

Interactive figures use the generated manifest to load appropriate rendering scripts:

// After loading both data.js and figure-manifest.js
const figureId = 'transformer-attention';
const providers = window.AIFS_FIGURE_PROVIDERS[figureId];

// Load the first available provider
if (providers && providers.length > 0) {
  const scriptSrc = providers[0];
  console.log(`Loading figure provider: ${scriptSrc}`);
}

Generating a Sitemap from SEO Manifests

For search engine optimization workflows, you can process the generated JSON server-side:

const fs = require('fs');
const seoData = JSON.parse(fs.readFileSync('site/lesson-seo.json', 'utf8'));

const sitemapEntries = Object.values(seoData.lessons).map(lesson => ({
  url: lesson.canonicalUrl,
  lastmod: new Date().toISOString(),
  priority: 0.8
}));

Summary

• The AI Engineering from Scratch website is generated by site/build.js, a Node.js script that transforms Markdown curriculum files into static assets.
• The build pipeline parses README.md, ROADMAP.md, and lesson documentation to create a canonical data structure exposed as window.AIFS_DATA.
• Interactive figures are resolved through discoverUsedFigureIds() and buildFigureProviderManifest(), which map figure IDs to specific JavaScript provider scripts.
• SEO metadata is extracted from lesson frontmatter and rendered into lesson-seo.json and certification-seo.json for search engine discoverability.
• The .github/workflows/curriculum.yml GitHub Action automates regeneration on every commit, ensuring the static site remains synchronized with repository content.
• All output artifacts—including data.js, HTML templates, and figure manifests—are pure static files requiring no server-side runtime.

Frequently Asked Questions

What triggers the website generation process?

A GitHub Actions workflow defined in .github/workflows/curriculum.yml monitors the main branch. On every push, the workflow executes node site/build.js, regenerating all static assets and committing the updated site/data.js back to the repository.

How does the build system handle interactive diagrams?

The script discoverUsedFigureIds() scans lesson Markdown for ```figure code fences to identify required visualizations. It then matches these IDs against provider scripts listed in FIGURE_PROVIDER_ORDER via buildFigureProviderManifest(), outputting the mapping to site/figure-manifest.js for client-side resolution.

Can I run the website generator locally without CI?

Yes. From the repository root, run npm install to install development dependencies, then execute node site/build.js. This regenerates site/data.js, SEO JSON files, and figure manifests without requiring the GitHub Actions environment.

What is the difference between site/data.js and the SEO JSON files?

site/data.js is a JavaScript module that defines window.AIFS_DATA, containing the complete curriculum hierarchy, lesson metadata, and navigation structure required by the site UI. The SEO JSON files (lesson-seo.json and certification-seo.json) contain search-engine optimization metadata—titles, descriptions, keywords, and canonical URLs—used for generating HTML meta tags and sitemaps.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}