# How site/build.js Generates site/data.js from the AI Curriculum in ai-engineering-from-scratch

> Discover how site/build.js in ai-engineering-from-scratch generates site/data.js by parsing markdown files to create static JavaScript constants for the documentation website.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-20

---

**The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script in `rohitg00/ai-engineering-from-scratch` parses markdown curriculum files (README.md, ROADMAP.md, and lesson documentation) to generate a static [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) bundle containing three serialized JavaScript constants—`PHASES`, `GLOSSARY`, and `ARTIFACTS`—that power the documentation website.**

The `rohitg00/ai-engineering-from-scratch` repository uses a custom Node.js build pipeline to transform its textual curriculum into a machine-readable JavaScript data file. This automated process ensures the static website always reflects the latest lesson statuses, glossary definitions, and reusable code artefacts without requiring manual JSON updates.

## Build Pipeline Architecture

The generation process follows a deterministic seven-step pipeline defined in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js). The script establishes absolute paths to key curriculum files, then orchestrates a series of parsing functions that extract structured data from markdown sources.

```js
const REPO_ROOT = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');
const OUTPUT_PATH = path.join(__dirname, 'data.js');

```

*Lines 14-19 in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) define the canonical file paths that the build script consumes.*

## Parsing ROADMAP.md for Status Tracking

The `parseRoadmap()` function scans [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to extract per-phase and per-lesson completion statuses. It detects phase headers using patterns like `## Phase X … — ✅|🚧|⬚` and parses lesson table rows to map status emojis to structured states.

```js
function parseRoadmap(content) {
  const statuses = {};
  // …iterates line‑by‑line, detecting “## Phase X … — ✅|🚧|⬚” headers

  // and “| 01 | Dev Environment | ✅ |” lesson rows
}

```

This function returns an object keyed by `"Phase <id>"` containing the phase status and a nested map of lesson names to their respective states (`complete`, `in-progress`, or `planned`). The implementation occupies lines 31-61 in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js).

## Extracting Phase and Lesson Metadata from README.md

The `parseReadme()` function (lines 63-130) processes the master curriculum list to identify phases, lessons, types, and GitHub URLs. It detects phase headers using both legacy and new badge formats, then extracts lesson tables to capture:

- Lesson name and type (e.g., "Build", "Theory")
- Programming languages (Python, TypeScript, Rust)
- GitHub repository URLs
- Initial status derived from the roadmap data

The function constructs an array of phase objects, each containing an ordered list of lesson objects that serve as the foundation for the `PHASES` constant.

## Enriching Lessons with Summaries and Keywords

After establishing the lesson hierarchy, the script calls `extractLessonMeta()` for each lesson that contains a GitHub URL. This function reads the lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file to extract human-readable metadata.

```js
function extractLessonMeta(relPath) {
  const docPath = path.join(REPO_ROOT, relPath, 'docs', 'en.md');
  // Reads the file, grabs the first “> …” blockquote as `summary`
  // Collects all H3 headings as `keywords`
}

```

The implementation (lines 33-68) captures the first blockquote as the lesson summary and aggregates all H3 headings as searchable keywords. The main loop (lines 41-49) attaches these `summary` and `keywords` properties to each lesson object, enriching the curriculum data with descriptive content for the website's search functionality.

## Building the Glossary Index

The `parseGlossary()` function (lines 71-100) processes [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) to create a searchable dictionary of AI terminology. It identifies terms by detecting `### Term` headings, then extracts definitions from specific markdown patterns:

- `**What people say:**` captures common usage
- `**What it actually means:**` captures technical definitions

The function produces an array of objects with the structure `{ term, says, means }`, which becomes the `GLOSSARY` constant in the output file.

## Discovering Reusable Artefacts

The `discoverArtifacts()` function (lines 40-98) traverses the `phases/*/*/outputs/` directory tree to identify reusable curriculum components. It extracts front-matter from markdown files and registers missions, skills, prompts, and agents as artefacts.

Each artefact record includes:
- `kind`: The artefact type (skill, prompt, agent, mission)
- `name` and `description`: Human-readable identifiers
- `tags`: Categorical metadata
- `phase` and `lesson`: Numeric indices for navigation
- `lessonPath` and `file`: Absolute and relative file paths

This discovery process ensures that practical code examples and templates are indexed separately from theoretical content.

## Serializing Output to data.js

The `build()` function (lines 68-78) orchestrates the final output generation. It aggregates the parsed data into three JavaScript constants and writes them to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) as a self-contained module:

```js
function build() {
  // …reads README, ROADMAP, glossary, discovers artefacts…
  const output = `// Auto‑generated by build.js — do not edit manually.
  // Last built: ${new Date().toISOString()}

  const PHASES = ${JSON.stringify(phases, null, 2)};

  const GLOSSARY = ${JSON.stringify(glossaryTerms, null, 2)};

  const ARTIFACTS = ${JSON.stringify(artifacts, null, 2)};
  `;
  fs.writeFileSync(OUTPUT_PATH, output, 'utf8');
}

```

The generated file uses literal JavaScript syntax rather than JSON, allowing the static site to load the data instantly via a `<script>` tag without requiring an additional fetch request. The script also updates README statistics, syncs badge counts, generates a sitemap, and creates an [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) file for AI agent consumption.

## Generated Data Structure

The resulting [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) contains three top-level constants that power the website's navigation and search:

```js
const PHASES = [
  {
    "id": 0,
    "name": "Setup & Tooling",
    "status": "complete",
    "desc": "Get your dev environment ready.",
    "lessons": [
      {
        "name": "Dev Environment",
        "status": "complete",
        "type": "Build",
        "lang": "Python, TypeScript, Rust",
        "url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/00-setup-and-tooling/01-dev-environment/",
        "summary": "Set up a reproducible Python/TS/Rust dev stack.",
        "keywords": "Virtualenv · Cargo · npm"
      }
    ]
  }
];

const GLOSSARY = [
  {
    "term": "Attention",
    "says": "Focus on the most relevant parts of the input.",
    "means": "A weighted sum of value vectors where weights come from queries and keys."
  }
];

const ARTIFACTS = [
  {
    "kind": "skill",
    "name": "Prompt‑Engineer",
    "description": "A reusable prompt template for instruction‑following models.",
    "tags": ["prompt", "template"],
    "phase": 7,
    "lesson": 3,
    "lessonPath": "phases/07-foundations/03-prompt-engineer",
    "file": "phases/07-foundations/03-prompt-engineer/outputs/skill-prompt-engineer.md"
  }
];

```

The literal JavaScript format ensures immediate availability upon page load, supporting the site's client-side rendering without asynchronous data fetching.

## Automating the Build Process

The repository automates [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) execution through a GitHub Actions workflow defined in [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml). The workflow triggers on every push, ensuring [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) remains synchronized with the latest markdown curriculum changes.

For local development, regenerate the data file manually by running:

```bash
node site/build.js

```

This command updates [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) after editing any curriculum markdown, including README updates, lesson content changes, or glossary additions.

## Summary

- **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** serves as the single source of truth for curriculum data generation, parsing [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) to create a unified data bundle.
- The script produces three JavaScript constants—`PHASES`, `GLOSSARY`, and `ARTIFACTS`—that contain complete curriculum metadata, term definitions, and reusable code components.
- **Lesson enrichment** occurs through `extractLessonMeta()`, which pulls summaries from blockquotes and keywords from H3 headings in each lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file.
- **Artefact discovery** automatically indexes skills, prompts, and missions from `phases/*/*/outputs/` directories, making practical examples searchable.
- The generated **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** uses literal JavaScript syntax for immediate browser consumption, eliminating the need for runtime JSON parsing.
- The build process runs automatically via GitHub Actions on every push, or manually via `node site/build.js` for local development.

## Frequently Asked Questions

### How does site/build.js determine the completion status of each lesson?

The script parses [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) using the `parseRoadmap()` function to extract emoji-based status indicators. It maps ✅ (complete), 🚧 (in-progress), and ⬚ (planned) emojis found in phase headers and lesson table rows to standardized status strings (`complete`, `in-progress`, `planned`). These statuses are then attached to the corresponding lesson objects in the `PHASES` array during the README parsing phase.

### What file paths does the build script scan for lesson documentation?

The script looks for lesson-specific documentation at `phases/{phase}/{lesson}/docs/en.md` relative to the repository root. The `extractLessonMeta()` function constructs this path by combining the relative lesson path (extracted from the README's GitHub URLs) with the fixed [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) suffix. It then reads the first blockquote for the summary and collects all H3 headings as keywords.

### Can I run the build script locally without triggering the GitHub Actions workflow?

Yes. Run `node site/build.js` from the repository root to manually regenerate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). This executes the same parsing logic used in the CI pipeline, updating the data file with any local changes you've made to [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), lesson documentation, or glossary terms. The script will overwrite [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) with fresh serialized data containing the current timestamp.

### What is the difference between the PHASES and ARTIFACTS constants in data.js?

The `PHASES` constant contains the hierarchical curriculum structure—phases and lessons with their metadata, URLs, and completion statuses—essentially representing the learning path. The `ARTIFACTS` constant indexes reusable components discovered in lesson `outputs/` folders, such as prompt templates, agent configurations, and skill definitions. While `PHASES` organizes content for human navigation, `ARTIFACTS` provides machine-readable references to practical code components that developers can reuse in their own AI engineering projects.