How site/build.js Generates site/data.js from Curriculum Data in AI Engineering from Scratch

The site/build.js script parses human-readable Markdown curriculum files—specifically README.md, ROADMAP.md, glossary/terms.md, and individual lesson documentation—to generate site/data.js, a machine-readable JavaScript module that exports PHASES, GLOSSARY, and ARTIFACTS constants for the frontend to consume.

The ai-engineering-from-scratch repository by rohitg00 maintains a single-source-of-truth workflow where authors edit intuitive Markdown files. The site/build.js utility transforms these static source files into structured data, eliminating manual duplication and ensuring the curriculum website always reflects the latest repository state. This build process runs automatically on every push via GitHub Actions and can be triggered manually with a single command.

The Build Pipeline Architecture

The script executes a nine-step pipeline to aggregate curriculum data. At lines 14–19 of site/build.js, the script first defines constants pointing to the repository root and input files: README_PATH, ROADMAP_PATH, GLOSSARY_PATH, and OUTPUT_PATH (targeting site/data.js).

Then it sequentially parses each curriculum source:

1. ROADMAP.md processing extracts completion status via parseRoadmap() (lines 30–61)
2. README.md processing extracts lesson structure via parseReadme() (lines 63–130)
3. glossary/terms.md processing extracts term definitions via parseGlossary() (lines 71–109)
4. Artifact discovery scans output directories via discoverArtifacts() (lines 40–97)
5. Lesson enrichment fetches metadata from lesson docs via extractLessonMeta() (lines 33–68)
6. Stats calculation computes totals for phases, lessons, and terms (lines 52–66)
7. File generation writes the final site/data.js (lines 68–78)
8. Side effects update badges, sitemaps, and build metadata

Parsing Curriculum Sources

ROADMAP.md Status Extraction

The parseRoadmap() function (lines 30–61) scans the roadmap table to extract each phase’s overall status and the status of every individual lesson. It identifies visual markers (✅, 🚧, ⬚) to determine completion states. The function returns a nested object mapping phase numbers to their status and lesson completion dictionaries:

{
  "Phase 1": {
    phaseStatus: "complete",
    lessons: {
      "Lesson Name": "complete",
      "Another Lesson": "in-progress"
    }
  }
}

README.md Structure Parsing

The parseReadme() function (lines 63–130) walks the README line-by-line, detecting phase headers and lesson tables. For each lesson discovered, it extracts:

• name: The lesson title
• type: Build or Learn classification
• lang: Programming language (Python, TypeScript, Rust, etc.)
• url: GitHub repository link
• status: Injected from the roadmap data (fallback to "planned")

This function constructs the phases array containing objects with id, name, status, desc, and the lessons array.

Glossary Term Extraction

The parseGlossary() function (lines 71–109) processes glossary/terms.md to extract every term defined under ### Term headings. It captures the "What people say" and "What it actually means" blocks, returning an array of objects structured as {term, says, means}.

Enriching Lesson Metadata

Discovering Artifacts

The discoverArtifacts() function (lines 40–97) walks the phases/**/outputs directories to find reusable curriculum outputs. It reads front-matter from files matching skill-*.md, prompt-*.md, or agent-*.md patterns, plus any mission.md files. The function builds an artifacts array containing metadata objects with:

{
  kind: "skill",
  name: "Linear regression skill",
  description: "A tiny demo of gradient descent.",
  tags: ["regression", "gradient-descent"],
  phase: 2,
  lesson: 5,
  lessonPath: "phases/02-math/05-linear-regression",
  file: "phases/02-math/05-linear-regression/outputs/skill-linear-regression.md"
}

Extracting Lesson Summaries

For every lesson with a GitHub URL, extractLessonMeta() (lines 33–68) reads the corresponding docs/en.md file. It extracts the first blockquote as a summary and all ### headings as keywords. These fields are added directly to the lesson objects in the phases array, enabling the frontend to display lesson previews without additional network requests.

Generating the Output File

The final generation block (lines 68–78) constructs site/data.js by serializing three JavaScript constants:

const PHASES = [/* ... curriculum hierarchy ... */];
const GLOSSARY = [/* ... term definitions ... */];
const ARTIFACTS = [/* ... reusable outputs ... */];

The script uses JSON.stringify(..., null, 2) for readable formatting, wrapping the serialized data in const declarations. This creates a valid JavaScript module that the frontend imports directly.

Running the Build Locally

Execute the build from the repository root to generate site/data.js manually:

node site/build.js

The script outputs progress to the console:

📖 Reading source files...
🔍 Parsing ROADMAP.md...
🔍 Parsing README.md...
🔍 Parsing glossary/terms.md...
🔍 Discovering outputs + Phase 14 missions...
📚 Extracting lesson summaries + keywords from docs/en.md...
📊 Stats:
   Phases: 19
   Lessons: 435
   Complete: 312
   Summaries: 280, Keywords: 254
✅ Generated site/data.js

Generated Data Structure

The output file site/data.js exposes three globally available constants when imported:

// Auto-generated by build.js — do not edit manually.
const PHASES = [
  {
    "id": 0,
    "name": "Setup & Tooling",
    "status": "complete",
    "desc": "Get your dev environment ready.",
    "lessons": [
      {
        "name": "Dev Environment",
        "status": "complete",
        "type": "Build",
        "lang": "Python, Node, Rust",
        "url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/00-setup-and-tooling/01-dev-environment/",
        "summary": "Install Python, Node, Rust …",
        "keywords": "Virtualenv · npm · cargo"
      }
    ]
  }
];

const GLOSSARY = [
  {
    "term": "Agent",
    "says": "A thing that perceives…",
    "means": "An autonomous system that ..."
  }
];

const ARTIFACTS = [
  {
    "kind": "skill",
    "name": "Linear regression skill",
    "description": "A tiny demo of gradient descent.",
    "tags": ["regression","gradient-descent"],
    "phase": 2,
    "lesson": 5,
    "lessonPath": "phases/02-math/05-linear-regression",
    "file": "phases/02-math/05-linear-regression/outputs/skill-linear-regression.md"
  }
];

The frontend imports this module to render the lesson catalog, searchable glossary, and artifact library without server-side processing.

Summary

• Single command execution: Running node site/build.js from the repository root aggregates all curriculum data into site/data.js.
• Three primary parsers: parseRoadmap(), parseReadme(), and parseGlossary() extract structure and status from ROADMAP.md, README.md, and glossary/terms.md respectively.
• Deep metadata enrichment: The extractLessonMeta() function pulls summaries and keywords from individual lesson docs/en.md files.
• Artifact discovery: The discoverArtifacts() function scans phases/**/outputs/*.md files to build a reusable output library.
• Machine-readable output: The final site/data.js exports three constants—PHASES, GLOSSARY, and ARTIFACTS—formatted with JSON.stringify for immediate consumption by the frontend.

Frequently Asked Questions

What source files does site/build.js read to generate data.js?

The script reads README.md for lesson structure, ROADMAP.md for completion status, and glossary/terms.md for term definitions. It also traverses the phases/ directory to locate docs/en.md files for lesson metadata and outputs/ directories for artifacts (skill, prompt, and agent files).

How are lesson completion statuses determined?

The parseRoadmap() function scans ROADMAP.md for visual status markers—specifically ✅ (complete), 🚧 (in-progress), and ⬚ (planned)—and maps these to string values like "complete", "in-progress", or "planned". These statuses are then injected into the corresponding lesson objects during parseReadme() execution.

Can I run the build script locally without GitHub Actions?

Yes. Execute node site/build.js from the repository root. The script requires Node.js and filesystem access to read the Markdown sources and write site/data.js. It performs the same aggregation that runs in the CI pipeline, printing progress and statistics to the console upon completion.

What is the structure of the generated site/data.js file?

The file is a valid JavaScript module containing three exported constants: PHASES (an array of phase objects with nested lessons), GLOSSARY (an array of term definitions with "says" and "means" fields), and ARTIFACTS (an array of reusable outputs with metadata like kind, tags, and file paths). All data is serialized using JSON.stringify with 2-space indentation for readability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}