# How site/build.js Generates site/data.js from Curriculum Data in AI Engineering from Scratch

> Discover how site/build.js transforms curriculum Markdown files into site/data.js for the AI Engineering from Scratch project. Understand the data generation process.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-24

---

**The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script parses human-readable Markdown curriculum files—specifically [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md), and individual lesson documentation—to generate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), a machine-readable JavaScript module that exports `PHASES`, `GLOSSARY`, and `ARTIFACTS` constants for the frontend to consume.**

The `ai-engineering-from-scratch` repository by rohitg00 maintains a single-source-of-truth workflow where authors edit intuitive Markdown files. The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) utility transforms these static source files into structured data, eliminating manual duplication and ensuring the curriculum website always reflects the latest repository state. This build process runs automatically on every push via GitHub Actions and can be triggered manually with a single command.

## The Build Pipeline Architecture

The script executes a nine-step pipeline to aggregate curriculum data. At lines 14–19 of [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js), the script first defines constants pointing to the repository root and input files: `README_PATH`, `ROADMAP_PATH`, `GLOSSARY_PATH`, and `OUTPUT_PATH` (targeting [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)).

Then it sequentially parses each curriculum source:

1. **ROADMAP.md** processing extracts completion status via `parseRoadmap()` (lines 30–61)
2. **README.md** processing extracts lesson structure via `parseReadme()` (lines 63–130)
3. **glossary/terms.md** processing extracts term definitions via `parseGlossary()` (lines 71–109)
4. **Artifact discovery** scans output directories via `discoverArtifacts()` (lines 40–97)
5. **Lesson enrichment** fetches metadata from lesson docs via `extractLessonMeta()` (lines 33–68)
6. **Stats calculation** computes totals for phases, lessons, and terms (lines 52–66)
7. **File generation** writes the final [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) (lines 68–78)
8. **Side effects** update badges, sitemaps, and build metadata

## Parsing Curriculum Sources

### ROADMAP.md Status Extraction

The `parseRoadmap()` function (lines 30–61) scans the roadmap table to extract each phase’s overall status and the status of every individual lesson. It identifies visual markers (✅, 🚧, ⬚) to determine completion states. The function returns a nested object mapping phase numbers to their status and lesson completion dictionaries:

```javascript
{
  "Phase 1": {
    phaseStatus: "complete",
    lessons: {
      "Lesson Name": "complete",
      "Another Lesson": "in-progress"
    }
  }
}

```

### README.md Structure Parsing

The `parseReadme()` function (lines 63–130) walks the README line-by-line, detecting phase headers and lesson tables. For each lesson discovered, it extracts:

- **name**: The lesson title
- **type**: Build or Learn classification
- **lang**: Programming language (Python, TypeScript, Rust, etc.)
- **url**: GitHub repository link
- **status**: Injected from the roadmap data (fallback to "planned")

This function constructs the `phases` array containing objects with `id`, `name`, `status`, `desc`, and the `lessons` array.

### Glossary Term Extraction

The `parseGlossary()` function (lines 71–109) processes [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) to extract every term defined under `### Term` headings. It captures the "What people say" and "What it actually means" blocks, returning an array of objects structured as `{term, says, means}`.

## Enriching Lesson Metadata

### Discovering Artifacts

The `discoverArtifacts()` function (lines 40–97) walks the `phases/**/outputs` directories to find reusable curriculum outputs. It reads front-matter from files matching `skill-*.md`, `prompt-*.md`, or `agent-*.md` patterns, plus any [`mission.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/mission.md) files. The function builds an `artifacts` array containing metadata objects with:

```javascript
{
  kind: "skill",
  name: "Linear regression skill",
  description: "A tiny demo of gradient descent.",
  tags: ["regression", "gradient-descent"],
  phase: 2,
  lesson: 5,
  lessonPath: "phases/02-math/05-linear-regression",
  file: "phases/02-math/05-linear-regression/outputs/skill-linear-regression.md"
}

```

### Extracting Lesson Summaries

For every lesson with a GitHub URL, `extractLessonMeta()` (lines 33–68) reads the corresponding [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file. It extracts the first blockquote as a **summary** and all `###` headings as **keywords**. These fields are added directly to the lesson objects in the phases array, enabling the frontend to display lesson previews without additional network requests.

## Generating the Output File

The final generation block (lines 68–78) constructs [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) by serializing three JavaScript constants:

```javascript
const PHASES = [/* ... curriculum hierarchy ... */];
const GLOSSARY = [/* ... term definitions ... */];
const ARTIFACTS = [/* ... reusable outputs ... */];

```

The script uses `JSON.stringify(..., null, 2)` for readable formatting, wrapping the serialized data in `const` declarations. This creates a valid JavaScript module that the frontend imports directly.

## Running the Build Locally

Execute the build from the repository root to generate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) manually:

```bash
node site/build.js

```

The script outputs progress to the console:

```text
📖 Reading source files...
🔍 Parsing ROADMAP.md...
🔍 Parsing README.md...
🔍 Parsing glossary/terms.md...
🔍 Discovering outputs + Phase 14 missions...
📚 Extracting lesson summaries + keywords from docs/en.md...
📊 Stats:
   Phases: 19
   Lessons: 435
   Complete: 312
   Summaries: 280, Keywords: 254
✅ Generated site/data.js

```

## Generated Data Structure

The output file [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) exposes three globally available constants when imported:

```javascript
// Auto-generated by build.js — do not edit manually.
const PHASES = [
  {
    "id": 0,
    "name": "Setup & Tooling",
    "status": "complete",
    "desc": "Get your dev environment ready.",
    "lessons": [
      {
        "name": "Dev Environment",
        "status": "complete",
        "type": "Build",
        "lang": "Python, Node, Rust",
        "url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/00-setup-and-tooling/01-dev-environment/",
        "summary": "Install Python, Node, Rust …",
        "keywords": "Virtualenv · npm · cargo"
      }
    ]
  }
];

const GLOSSARY = [
  {
    "term": "Agent",
    "says": "A thing that perceives…",
    "means": "An autonomous system that ..."
  }
];

const ARTIFACTS = [
  {
    "kind": "skill",
    "name": "Linear regression skill",
    "description": "A tiny demo of gradient descent.",
    "tags": ["regression","gradient-descent"],
    "phase": 2,
    "lesson": 5,
    "lessonPath": "phases/02-math/05-linear-regression",
    "file": "phases/02-math/05-linear-regression/outputs/skill-linear-regression.md"
  }
];

```

The frontend imports this module to render the lesson catalog, searchable glossary, and artifact library without server-side processing.

## Summary

- **Single command execution**: Running `node site/build.js` from the repository root aggregates all curriculum data into [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js).
- **Three primary parsers**: `parseRoadmap()`, `parseReadme()`, and `parseGlossary()` extract structure and status from [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) respectively.
- **Deep metadata enrichment**: The `extractLessonMeta()` function pulls summaries and keywords from individual lesson [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files.
- **Artifact discovery**: The `discoverArtifacts()` function scans `phases/**/outputs/*.md` files to build a reusable output library.
- **Machine-readable output**: The final [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) exports three constants—`PHASES`, `GLOSSARY`, and `ARTIFACTS`—formatted with `JSON.stringify` for immediate consumption by the frontend.

## Frequently Asked Questions

### What source files does site/build.js read to generate data.js?

The script reads [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) for lesson structure, [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for completion status, and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) for term definitions. It also traverses the `phases/` directory to locate [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files for lesson metadata and `outputs/` directories for artifacts (skill, prompt, and agent files).

### How are lesson completion statuses determined?

The `parseRoadmap()` function scans [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for visual status markers—specifically ✅ (complete), 🚧 (in-progress), and ⬚ (planned)—and maps these to string values like "complete", "in-progress", or "planned". These statuses are then injected into the corresponding lesson objects during `parseReadme()` execution.

### Can I run the build script locally without GitHub Actions?

Yes. Execute `node site/build.js` from the repository root. The script requires Node.js and filesystem access to read the Markdown sources and write [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). It performs the same aggregation that runs in the CI pipeline, printing progress and statistics to the console upon completion.

### What is the structure of the generated site/data.js file?

The file is a valid JavaScript module containing three exported constants: `PHASES` (an array of phase objects with nested lessons), `GLOSSARY` (an array of term definitions with "says" and "means" fields), and `ARTIFACTS` (an array of reusable outputs with metadata like kind, tags, and file paths). All data is serialized using `JSON.stringify` with 2-space indentation for readability.