# How site/build.js Generates data.js from README.md and ROADMAP.md in ai-engineering-from-scratch

> Discover how site/build.js in ai-engineering-from-scratch generates data.js by parsing README.md and ROADMAP.md, enriching data with metadata and glossary terms to power the curriculum website.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-22

---

**The site/build.js script reads README.md and ROADMAP.md, parses their markdown tables and status indicators, enriches the data with lesson metadata and glossary terms, and exports a JavaScript module to site/data.js that powers the curriculum website.**

The ai-engineering-from-scratch repository uses a custom build pipeline to transform static markdown documentation into a structured JavaScript data module. Understanding how site/build.js generates data.js from README.md and ROADMAP.md reveals how the curriculum site maintains synchronization with the source documentation without manual data entry. This automated process ensures that lesson statuses, phase structures, and glossary definitions always reflect the current state of the repository.

## Build Pipeline Overview

The build system orchestrates multiple parsing stages to convert human-readable markdown into machine-readable JavaScript constants. Located at [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js), the script executes an eight-step pipeline that processes curriculum files, extracts metadata, and writes the final output to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). Each function handles a specific data transformation, from parsing roadmap status indicators to discovering reusable artifacts across lesson directories.

## Step 1: Loading Source Files

The script begins by resolving absolute paths to the repository root and target markdown files. According to lines 23–26 of [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js), it defines constants for the three primary inputs:

```javascript
const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

```

These paths enable the build system to read [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) for curriculum structure, [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for completion status, and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) for domain definitions.

## Step 2: Parsing ROADMAP.md for Lesson Status

The `parseRoadmap()` function (lines 30–61) processes the roadmap file to extract phase headings and lesson completion states. It scans each line to identify:

- **Phase headings** marked with status emojis (✅ for complete, 🚧 for in-progress, ⬚ for planned)
- **Lesson rows** formatted as markdown tables (`| 01 | Dev Environment | ✅ |`)

The function returns a nested object mapping phases to their statuses and lessons, creating a single source of truth for curriculum completion states.

## Step 3: Parsing README.md for Curriculum Structure

The `parseReadme()` function (lines 63–131) transforms the README's markdown tables into structured phase objects. It detects phase headers (both legacy "### Phase 0:" formats and newer badge styles) and initializes `currentPhase` objects with `id`, `name`, and `status` properties inherited from the roadmap data.

When encountering lesson tables (starting with `| # | Lesson …`), the parser extracts:

- **Lesson names** and optional URLs from the `lessonCol` field
- **Type classifications** (Build or Learn) by stripping badge images
- **Programming languages** through emoji-to-text conversion using the `EMOJI_LANG` mapping

Each lesson is matched against the roadmap data to assign `complete`, `in-progress`, or `planned` status. If a lesson includes a URL, the status is automatically set to `complete`.

## Step 4: Extracting Per-Lesson Metadata

For lessons containing URLs, the build script calls `extractLessonMeta()` (lines 33–68) to augment curriculum data with content summaries. This function reads each lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file and extracts:

- **Summary text** from the first blockquote (used as a one-line description)
- **Keywords** concatenated from all `###` headings within the lesson

The build loop (lines 39–49) integrates this metadata into the lesson objects, producing enriched entries containing `summary` and `keywords` fields alongside the core curriculum data.

## Step 5: Collecting Glossary Terms and Reusable Artifacts

The pipeline extends beyond curriculum content to include supplementary resources. The `parseGlossary()` function (lines 71–103) scans [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) for `###` headings and "What people say / What it actually means" lines to build the `GLOSSARY` array.

Simultaneously, `discoverArtifacts()` (lines 40–98) walks the `phases/` directory to locate reusable outputs:

- **Skill files** matching `outputs/skill-*.md`
- **Prompt files** matching `outputs/prompt-*.md`
- **Agent files** matching `outputs/agent-*.md`
- **Mission files** named [`mission.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/mission.md) within lesson directories

Each artifact's frontmatter is parsed to populate the `ARTIFACTS` export.

## Step 6: Generating the Final data.js File

After aggregating `phases`, `glossaryTerms`, and `artifacts`, the script constructs the final output string. Lines 69–78 write a JavaScript module containing three exported constants:

```javascript
const PHASES   = [...];   // full curriculum tree
const GLOSSARY = [...];   // glossary entries
const ARTIFACTS = [...];  // reusable outputs

```

This content is written to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) with an auto-generated comment header warning against manual edits. The resulting file serves as the data layer for the curriculum website, consumed by downstream components to render the dynamic interface.

## Running the Build Locally

Execute the build script from the repository root to regenerate the data module:

```bash
node site/build.js

```

The script prints progress logs and recreates [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) with the latest curriculum state. Import the generated module in your application to access the structured data:

```javascript
import { PHASES, GLOSSARY, ARTIFACTS } from './site/data.js';

// List all completed lessons
const completed = PHASES.flatMap(p => p.lessons)
                       .filter(l => l.status === 'complete')
                       .map(l => l.name);
console.log('Completed lessons:', completed);

```

Inspect the generated structure to verify the output:

```bash
head -n 20 site/data.js

```

The output begins with an auto-generated warning followed by the `PHASES` array containing the complete curriculum hierarchy.

## Summary

- **site/build.js** serves as the central build pipeline that transforms markdown documentation into a JavaScript data module.
- The script parses **ROADMAP.md** (lines 30–61) to extract phase and lesson completion statuses using emoji indicators.
- **README.md** parsing (lines 63–131) extracts curriculum structure, lesson types, and programming languages from markdown tables.
- Per-lesson metadata is enriched by reading [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files via `extractLessonMeta()` to capture summaries and keywords.
- The final [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) exports three constants—`PHASES`, `GLOSSARY`, and `ARTIFACTS`—that power the curriculum website.
- Running `node site/build.js` locally regenerates the data file, ensuring the site always reflects the current repository state.

## Frequently Asked Questions

### Where does site/build.js look for source files?

The script resolves paths relative to its own directory, looking for [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) in the repository root. It defines these paths using `path.resolve(__dirname, '..')` to ensure cross-platform compatibility.

### How does the build script determine if a lesson is complete?

The script checks [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for status emojis (✅, 🚧, or ⬚) in the lesson table rows. Additionally, if a lesson entry in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) contains a URL field, the `parseReadme()` function automatically assigns `complete` status regardless of the roadmap indicator.

### Can I modify data.js manually?

No. The file header explicitly states that [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) is auto-generated by [`build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build.js) and should not be edited manually. Any changes would be overwritten the next time the build script runs, either locally or via the GitHub Actions workflow.

### What data structures are exported from data.js?

The generated module exports three constants: `PHASES` (an array of curriculum phases containing nested lesson objects), `GLOSSARY` (an array of term definitions), and `ARTIFACTS` (an array of reusable skills, prompts, and agents with their frontmatter metadata).