# How site/build.js Generates data.js from README.md and ROADMAP.md

> Discover how site/build.js in ai-engineering-from-scratch generates data.js by parsing markdown tables from README.md and ROADMAP.md, creating structured phase and lesson data.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-21

---

**TLDR:** The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script in the `rohitg00/ai-engineering-from-scratch` repository parses markdown tables from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to extract phase and lesson structures, enriches them with per-lesson metadata and glossary terms, and writes a static JavaScript module to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) that exports `PHASES`, `GLOSSARY`, and `ARTIFACTS` arrays for the curriculum site.

The static site powering the AI Engineering From Scratch curriculum doesn't rely on a traditional CMS. Instead, [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) generates [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js) from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to create a single source of truth where documentation changes automatically synchronize with the website's navigation and progress tracking.

## Loading the Source Documentation

The build process starts by resolving paths relative to the repository root. In [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js), the script defines constants pointing to the primary curriculum files (lines 23‑26):

```js
const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

```

These paths enable the script to read the human-readable curriculum definition alongside the glossary terms located in [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md).

## Parsing ROADMAP.md for Lesson Status

The **`parseRoadmap()`** function (lines 30‑61) serves as the single source of truth for lesson completion status. It scans [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) line-by-line to extract:

- **Phase headings** marked with status emojis: ✅ (complete), 🚧 (in‑progress), or ⬚ (planned)
- **Lesson rows** from markdown tables formatted as `│ 01 │ Dev Environment │ ✅ │`

The function returns a nested object mapping each phase to its status and a dictionary of lesson names to their respective statuses (`complete`, `in‑progress`, or `planned`).

## Extracting Curriculum Structure from README.md

While [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) tracks status, **`parseReadme()`** (lines 63‑131) extracts the hierarchical curriculum structure from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md). This function identifies phase headers—supporting both the legacy "### Phase 0:" format and newer badge-style headers—and initializes `currentPhase` objects containing `id`, `name`, and `status` pulled from the roadmap data.

When the parser encounters lesson tables (detecting rows starting with `│ # │ Lesson`), it extracts:

- **Lesson name** and optional URL from the lesson column
- **Lesson type** (Build or Learn) by stripping badge images
- **Programming languages** by converting emoji flags to plain text using the internal `EMOJI_LANG` mapping

Each lesson object receives its status by matching against the roadmap data, with one exception: lessons containing a URL are automatically forced to `complete` status. The completed lesson objects are pushed into `currentPhase.lessons` arrays, ultimately returning the full `phases` array.

## Enriching Lessons with Metadata

After establishing the curriculum tree, the build loop (lines 39‑49) walks every lesson that has a defined URL and calls **`extractLessonMeta()`** (lines 33‑68). This function reads each lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file to extract:

- A **summary** from the first blockquote (one-liner description)
- **Keywords** concatenated from all `###` headings within the lesson document

These fields populate the `summary` and `keywords` properties on each lesson object, enabling search and preview functionality on the frontend.

## Aggregating Glossary and Reusable Artifacts

Beyond the core curriculum, the script aggregates supplementary content through two dedicated functions:

**`parseGlossary()`** (lines 71‑103) scans [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) for `###` headings and extracts "What people say / What it actually means" pairs, returning an array of term objects that become the `GLOSSARY` export.

**`discoverArtifacts()`** (lines 40‑98) walks the `phases/` directory to discover reusable outputs. It identifies files in `outputs/` subdirectories matching patterns `skill-*.md`, `prompt-*.md`, or `agent-*.md`, parses their frontmatter, and includes any [`mission.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/mission.md) files found in lesson directories. These are compiled into the `ARTIFACTS` export for quick reference.

## Writing the data.js Module

With `phases`, `glossaryTerms`, and `artifacts` collected, the script constructs the final output string. It writes to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) (lines 69‑78) a JavaScript module prefixed with an auto-generation warning comment:

```js
// Auto-generated by build.js — do not edit manually.
const PHASES = [...];
const GLOSSARY = [...];
const ARTIFACTS = [...];

```

This file serves as the static data layer for the curriculum website, consumed directly by the frontend to render phase navigation, lesson lists, and search indexes.

## Auxiliary Build Tasks

The **`build()`** function (lines 19‑86) orchestrates additional synchronization tasks executed at build time:

- **`syncReadme()`**: Updates completion badge counts in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md)
- **`syncCounts()`**: Synchronizes marketing numbers across documentation
- **`writeSitemap()`**: Generates XML sitemaps for SEO
- **LLMs.txt generation**: Creates an AI-friendly digest of the curriculum
- **Build metadata**: Stores the current Git ref in [`build-meta.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build-meta.js)

The entire pipeline executes when the script runs via `build();` on line 601, typically triggered by GitHub Actions on every commit.

## Running the Build Locally

Execute the generator from the repository root to regenerate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js):

```bash
node site/build.js

```

The script prints progress logs and overwrites [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) with the latest curriculum state derived from the current markdown files.

## Consuming the Generated Data

Import the generated constants in frontend components or Node.js scripts:

```js
import { PHASES, GLOSSARY, ARTIFACTS } from './site/data.js';

// Filter for completed Python lessons
const pythonLessons = PHASES.flatMap(p => p.lessons)
  .filter(l => l.status === 'complete' && l.lang.includes('Python'));

```

## Summary

- **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** transforms markdown documentation into structured JavaScript by parsing [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) for curriculum hierarchy and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for completion status
- The **`parseRoadmap()`** and **`parseReadme()`** functions extract phase structures, lesson metadata, and language mappings from markdown tables
- **`extractLessonMeta()`** enriches lessons with summaries and keywords by reading individual lesson [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files
- The script aggregates glossary terms from [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) and reusable artifacts from `phases/` subdirectories via **`parseGlossary()`** and **`discoverArtifacts()`**
- Output is written to **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** as static exports (`PHASES`, `GLOSSARY`, `ARTIFACTS`) consumed by the curriculum website

## Frequently Asked Questions

### How does the build script determine a lesson's completion status?

The script checks **[`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md)** for emoji indicators (✅ for complete, 🚧 for in-progress, ⬚ for planned) matched to lesson names. Additionally, any lesson with a defined URL in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) is automatically forced to `complete` status regardless of the roadmap indicator.

### Can I modify the generated data.js file manually?

No. The file header explicitly states `// Auto-generated by build.js — do not edit manually.` Any manual changes will be overwritten the next time `node site/build.js` executes, which typically occurs automatically via GitHub Actions on every repository push.

### What markdown structure does the parser expect for lesson tables?

The script expects standard GitHub-flavored markdown tables with columns for lesson number, name, type (Build/Learn badges), and languages (emoji flags). The parser specifically looks for rows starting with `│` (pipe) characters and extracts cells based on the `│ # │ Lesson` header pattern found in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md).

### Where does the glossary content originate?

Glossary entries are sourced from **[`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md)**, where the **`parseGlossary()`** function extracts `###` headings as terms and parses "What people say / What it actually means" lines as definitions. This content is exported as the `GLOSSARY` constant in [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js).