# How the Site Build System Generates data.js from README and ROADMAP in AI Engineering From Scratch

> Discover how the ai-engineering-from-scratch build system uses Node.js to generate data.js from README and ROADMAP markdown files, extracting key project information.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-17

---

**The site build system in `ai-engineering-from-scratch` uses a Node.js script at [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) to parse structured markdown tables from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), extracting lesson metadata, phase statuses, and glossary terms to auto-generate the static [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) module that powers the curriculum website.**

The `ai-engineering-from-scratch` repository maintains its curriculum as human-readable markdown documentation. Instead of manually maintaining a separate data file, the project uses an automated pipeline that transforms these markdown sources into a machine-readable JavaScript module. This approach ensures the website curriculum stays perfectly synchronized with the repository's learning roadmap.

## Overview of the Build Pipeline

The core build logic lives in **[[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** and executes automatically via GitHub Actions or manually via Node.js. The script begins by resolving paths to the repository root and loading three primary source files:

```javascript
const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

```

When invoked, the `build()` function (lines 19‑86) orchestrates a multi-stage pipeline that reads these files, parses their structured content, enriches the data with additional metadata, and writes the final output to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js).

## Parsing ROADMAP.md for Lesson Status

The **`parseRoadmap()`** function (lines 30‑61) serves as the single source of truth for lesson completion status. It processes [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) line-by-line to extract phase headings and lesson rows.

The function identifies:
- **Phase status** via emoji indicators: ✅ = complete, 🚧 = in-progress, ⬚ = planned
- **Lesson entries** from table rows matching the pattern `| 01 | Dev Environment | ✅ |`

It returns a nested object structure mapping each phase to its status and lessons, enabling downstream functions to assign accurate status values to curriculum items.

## Extracting Curriculum Structure from README.md

The **`parseReadme()`** function (lines 63‑131) handles the heavy lifting of curriculum extraction from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md). This function supports both legacy "Phase N:" headers and newer badge-style formatting.

When the parser encounters a phase header, it initializes a `currentPhase` object with an `id`, `name`, and `status` (pulled from the roadmap data). It then detects the start of lesson tables using the pattern `| # | Lesson …` and extracts for each row:

- **Lesson name and URL** from the `lessonCol` column
- **Type classification** (Build vs. Learn) by stripping badge images
- **Programming languages** via emoji-to-text conversion using the `EMOJI_LANG` mapping

If a lesson includes a URL, the parser forces its status to `complete`. Otherwise, it matches the lesson name against the roadmap data to determine whether it is `complete`, `in-progress`, or `planned`. Each processed lesson becomes an entry in the `currentPhase.lessons` array.

## Enriching Data with Lesson Metadata

After establishing the curriculum structure, the build system augments each lesson with additional metadata. The **`extractLessonMeta()`** function (lines 33‑68) reads individual lesson files from their [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) paths to extract:

- **Summary**: The first blockquote found in the markdown file
- **Keywords**: Concatenated `###` headings from the lesson content

The main build loop (lines 39‑49) calls this function for every lesson that has a URL, adding `summary` and `keywords` properties to the lesson objects before they are added to the final phases array.

## Collecting Glossary and Artifacts

The build system consolidates auxiliary content through two additional parsers:

**`parseGlossary()`** (lines 71‑103) scans [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) for `###` headings and "What people say / What it actually means" lines, returning an array of term objects for the `GLOSSARY` export.

**`discoverArtifacts()`** (lines 40‑98) traverses the `phases/` directory searching for `outputs/*.md` files with specific prefixes (`skill-`, `prompt-`, `agent-`). It parses their frontmatter and also identifies [`mission.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/mission.md) files within lesson directories, building a flat `ARTIFACTS` list of reusable outputs.

## Generating the Final data.js File

With all data collected, the script constructs the final JavaScript module. It builds a string containing three exported constants:

```javascript
const PHASES   = [...];   // full curriculum tree
const GLOSSARY = [...];   // glossary entries
const ARTIFACTS = [...];  // reusable outputs

```

This string is written to **[[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** (lines 69‑78) with an auto-generated header comment warning against manual edits. The file also triggers auxiliary tasks including `syncReadme` (badge count updates), `syncCounts` (marketing numbers), `writeSitemap`, and [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) generation.

## Running the Build Locally

You can execute the build process manually to regenerate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) or inspect the output:

```bash

# From the repository root

node site/build.js

# → prints progress logs and (re)creates site/data.js

```

To inspect the generated structure:

```bash

# Show the first 20 lines of the generated file

head -n 20 site/data.js

```

The output begins with an auto-generated marker:

```javascript
// Auto-generated by build.js — do not edit manually.
const PHASES = [
  {
    "id": 0,
    "name": "Setup & Tooling",
    "status": "complete",
    // ...
  }
]

```

## Summary

- The **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** script is the central pipeline that transforms markdown documentation into JavaScript data structures.
- **`parseRoadmap()`** extracts lesson completion statuses (✅/🚧/⬚) from [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) tables.
- **`parseReadme()`** parses [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) for phase hierarchies, lesson URLs, types, and languages, then maps them to roadmap statuses.
- **`extractLessonMeta()`** enriches lessons with summaries and keywords from individual [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files.
- The final output at **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** exports three constants—`PHASES`, `GLOSSARY`, and `ARTIFACTS`—that power the entire curriculum website without manual data entry.

## Frequently Asked Questions

### What triggers the site build process?

The build executes automatically when GitHub Actions detects changes to the repository, or manually by running `node site/build.js` from the repository root. The workflow ensures [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) remains synchronized with any edits to [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) or [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md).

### How does the system determine if a lesson is complete?

The build system checks [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for emoji indicators (✅ for complete, 🚧 for in-progress, ⬚ for planned). Additionally, if `parseReadme()` detects a URL in the lesson table row, it automatically assigns `complete` status regardless of the roadmap marker.

### Can I manually edit the generated site/data.js file?

No. The file header explicitly states it is auto-generated by [`build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build.js). Manual edits would be overwritten during the next build cycle. Instead, modify the source files ([`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), or lesson [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md)) and rerun the build script.

### What other files does the build system generate?

Beyond [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), the `build()` function creates [`site/build-meta.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build-meta.js) (containing Git ref information), updates README badge counts via `syncReadme`, maintains marketing statistics through `syncCounts`, generates a sitemap via `writeSitemap`, and produces an [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) file for AI consumption.