# Data Generation Process for the AI Engineering Website: How `site/build.js` Works

> Explore the data generation process with site/build.js in the ai-engineering-from-scratch repository. Learn how this Node.js script transforms curriculum sources into a machine-readable JavaScript module.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-07-30

---

**The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script in the `rohitg00/ai-engineering-from-scratch` repository is a Node.js build pipeline that transforms human-editable curriculum sources into a machine-readable JavaScript module consumed by the static site.**

This script serves as the single source of truth for the AI Engineering from Scratch curriculum, parsing Markdown files, discovering reusable artifacts, and serializing everything into [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). The process ensures the website stays synchronized with every lesson, phase status, and glossary term defined in the repository.

## Overview of the Build Pipeline

The build process orchestrates twelve distinct stages, from parsing source documents to generating SEO-ready auxiliary files. When executed via `node site/build.js` (typically triggered by GitHub Actions on every push), the script performs a complete synthesis of the curriculum structure.

The pipeline begins by defining absolute paths to key source files. According to the source code in [Lines 14-21](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L14-L21), the script initializes constants pointing to [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md), and the output location [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js), along with the GitHub base URL used for constructing absolute lesson links.

## Parsing Source Documents

The script extracts structured data from three primary Markdown sources that serve as the curriculum's ground truth.

### Processing ROADMAP.md for Status Tracking

The `parseRoadmap` function ([Lines 30-61](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L30-L61)) scans the roadmap file to capture phase-level statuses and per-lesson completion emojis (✅, 🚧, ⬚). This produces a `roadmapStatuses` map that correlates lesson identifiers with their current implementation state, distinguishing between `complete`, `in-progress`, and `planned` lessons.

### Extracting Lesson Metadata from README.md

The `parseReadme` function ([Lines 63-132](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L63-L132)) walks through the main README to locate phase headers and lesson tables. For each lesson encountered, it extracts:

- **Lesson name** and **type** (Build, Theory, Capstone)
- **Language list** (Python, Node, Rust, etc.)
- **GitHub URL** for the lesson directory (when present)
- **Cross-referenced status** from the roadmap map

This function creates the foundational `PHASES` array that structures the entire curriculum navigation.

### Building the Glossary Index

The `parseGlossary` function ([Lines 71-103](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L71-L103)) reads [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) and extracts terminology definitions. For each term, it identifies the *what people say* and *what it actually means* sections, constructing an array of term objects used for the website's searchable glossary feature.

## Discovering Reusable Artifacts

A critical stage involves scanning the filesystem for curriculum artifacts using the `discoverArtifacts` function ([Lines 112-199](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L112-L199)).

This function recursively traverses every `phases/*/*/outputs` directory, searching for Markdown files following specific naming conventions:

- `skill-*.md` – Reusable skill implementations
- `prompt-*.md` – Standardized prompt templates
- `prompt-*.md` – Agent configurations

For each artifact found, the script extracts front-matter metadata, tags, and parent lesson references, yielding a flat `ARTIFACTS` array. This enables the UI to present reusable components alongside their originating lessons.

## Enriching Lesson Content

After establishing the basic structure, the build process enhances lesson objects with content-derived metadata. Within the `build()` function, a dedicated loop ([Lines 45-55](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L45-L55)) processes each lesson that has a valid URL:

1. **Reads** the lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file
2. **Extracts** the first blockquote as a one-line `summary`
3. **Parses** all `###` headings to compile `keywords`

This enrichment allows the website to display lesson previews and tag clouds without requiring runtime Markdown processing.

## Generating Output Files

### Serializing site/data.js

The core output occurs in [Lines 74-84](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L74-L84), where the script serializes three major data structures—`PHASES`, `GLOSSARY`, and `ARTIFACTS`—into a single JavaScript file. The generated [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) includes:

- A header comment containing the build timestamp
- The complete curriculum hierarchy with enriched metadata
- Searchable glossary definitions
- Reusable artifact registry

```javascript
// Auto-generated by build.js — do not edit manually.
// Last built: 2026-07-30T12:34:56.789Z

const PHASES = [
  {
    "id": 0,
    "name": "Setup & Tooling",
    "status": "complete",
    "desc": "Get your environment ready …",
    "lessons": [
      {
        "name": "Dev Environment",
        "status": "complete",
        "type": "Build",
        "lang": "Python, Node, Rust",
        "url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/00-setup-and-tooling/01-dev-environment/",
        "summary": "Configure a reproducible dev stack …",
        "keywords": "Virtualenv · Pip · Cargo · Rustup"
      }
    ]
  }
];

const GLOSSARY = [
  { "term":"Agent", "says":"A software entity that …", "means":"A program that …" }
];

const ARTIFACTS = [
  { "kind":"skill", "name":"Skill‑end‑to‑end‑safety‑gate", "description":"…", "tags":["safety"], "phase":19, "lesson":87 }
];

```

### Synchronizing Auxiliary Assets

The build pipeline updates several auxiliary files to maintain SEO and documentation consistency ([Lines 90-108](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L90-L108)):

- **README badges** – Updates completion counts and lesson statistics
- **Static HTML pages** – Rewrites count placeholders in template files
- **sitemap.xml** – Generates search engine sitemap entries for every lesson URL

```xml
<url>
  <loc>https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/01-dev-environment</loc>
  <lastmod>2026-07-30</lastmod>
  <changefreq>monthly</changefreq>
  <priority>0.6</priority>
</url>

```

### Creating Machine-Readable Indexes

The `writeLlms` function ([Lines 118-146](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L118-L146)) produces [`site/llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/llms.txt), a human-readable index of all lessons, glossary terms, and artifacts formatted for AI agent consumption.

## Build Metadata and Versioning

The script captures deployment context through the `resolveRef` function ([Lines 100-122](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L100-L122)), which determines the current git ref (preferring the `VERCEL_GIT_COMMIT_REF` environment variable when available). This ref is written to [`build-meta.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build-meta.js), enabling the client to fetch raw Markdown files from the correct branch or tag.

## Execution and Automation

The entry point ([Lines 606-608](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L606-L608)) invokes the `build()` function when the script runs directly. In production, GitHub Actions triggers this automatically on every push, ensuring the website reflects the latest curriculum state without manual intervention.

## Summary

- **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** serves as the central build orchestrator for the AI Engineering from Scratch curriculum, transforming Markdown sources into structured data.
- The pipeline parses **[`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md)** for status emojis, **[`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md)** for lesson metadata, and **[`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md)** for terminology definitions.
- **Artifact discovery** recursively scans `phases/*/*/outputs` directories to catalog reusable skills, prompts, and agents.
- The script enriches lesson objects by extracting summaries and keywords from individual [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files.
- Output is serialized to **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)**, accompanied by auxiliary files including [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml), [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt), and updated README badges.
- Build metadata captures the git ref via `VERCEL_GIT_COMMIT_REF` to ensure version-aligned content fetching.

## Frequently Asked Questions

### What is the role of [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) in the website architecture?

[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) is the runtime bundle consumed by the client-side application. It contains the serialized `PHASES`, `GLOSSARY`, and `ARTIFACTS` arrays generated by [`build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build.js), enabling static rendering of lesson navigation, search functionality, and glossary lookups without requiring server-side Markdown parsing.

### How does the build process track lesson completion status?

The `parseRoadmap` function scans [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for Unicode emojis (✅ for complete, 🚧 for in-progress, ⬚ for planned) and maps these to standardized status strings. When `parseReadme` processes lesson tables, it cross-references these mappings to assign each lesson its current implementation state.

### Where does the website get its lesson summaries and keywords?

During the build loop ([Lines 45-55](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js#L45-L55)), the script reads each lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file and extracts the first blockquote (treated as the summary) and all H3 headings (treated as keywords). This content is then embedded directly into the lesson objects within [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js).

### What is [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) and why is it generated?

[`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) is a machine-readable curriculum index created by the `writeLlms` function. It provides AI agents and large language models with a structured, link-rich overview of all lessons, glossary terms, and reusable artifacts, facilitating automated comprehension of the curriculum structure without requiring full repository traversal.