# Site Build Process for Generating data.js from README and ROADMAP Files

> Learn how to generate data.js from README and ROADMAP files using the site build process. This script extracts curriculum and lesson data to create a dynamic JavaScript file for your project.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-06-18

---

**The site build process uses [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) to parse [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), extracting curriculum structure and lesson statuses to generate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), a JavaScript module that powers the static curriculum site.**

The `rohitg00/ai-engineering-from-scratch` repository automates its curriculum website through a Node.js build pipeline. This system transforms human-readable markdown documentation into structured JavaScript data, eliminating manual synchronization between repository documentation and the deployed site.

## How the Build Script Reads Source Files

The build process begins in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) by resolving paths to the repository's core documentation files. At lines 23-26, the script defines constants pointing to [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) in the repository root:

```javascript
const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

```

These paths serve as the single source of truth for curriculum content, status tracking, and terminology definitions.

## Parsing ROADMAP.md for Lesson Status

The `parseRoadmap()` function (lines 30-61) scans [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to extract the completion status of every lesson and phase. It processes the file line-by-line to identify:

- **Phase headings** marked with status emojis (✅ for complete, 🚧 for in-progress, ⬚ for planned)
- **Lesson rows** formatted as tables (`| 01 | Dev Environment | ✅ |`)

The function returns a nested object mapping phase names to their status and lesson completion states, enabling the build to determine which lessons are `complete`, `in-progress`, or `planned` based on the emoji indicators present in the roadmap.

## Extracting Curriculum Structure from README.md

After establishing lesson statuses, the `parseReadme()` function (lines 63-131) processes [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) to build the curriculum hierarchy. This function:

1. **Detects phase headers** using both legacy "### Phase 0:" formats and newer badge-style headers

2. **Identifies lesson tables** by scanning for `| # | Lesson …` patterns

3. **Extracts lesson metadata** including:
   - **Lesson name** and optional URL from the lesson column
   - **Type** (Build or Learn) by stripping badge images
   - **Languages** by converting emoji flags to plain text using the `EMOJI_LANG` mapping

Each lesson object is matched against the roadmap data to assign its status. If a lesson includes a URL in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), the build forces its status to `complete` regardless of the roadmap indicator. The function returns a `phases` array containing phase objects with `id`, `name`, `status`, and a `lessons` array.

## Enriching Data with Per-Lesson Metadata

Once the basic structure is established, the build augments each lesson with detailed metadata using `extractLessonMeta()` (lines 33-68). For every lesson that has a URL, the function:

- Reads the corresponding [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file from the lesson directory
- Extracts the first blockquote as a one-line `summary`
- Concatenates all `###` headings to form a `keywords` string

The build loop (lines 39-49) iterates through all lessons, calling this extraction function and attaching the resulting `summary` and `keywords` properties to the lesson objects.

## Collecting Glossary Terms and Reusable Artifacts

The build aggregates supplementary content from two additional sources:

**Glossary Terms:** The `parseGlossary()` function (lines 71-103) scans [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) for `###` headings and "What people say / What it actually means" lines, returning an array of term objects for the `GLOSSARY` export.

**Reusable Artifacts:** The `discoverArtifacts()` function (lines 40-98) walks the `phases/` directory, searching for `outputs/*.md` files prefixed with `skill-`, `prompt-`, or `agent-`. It parses front-matter from these files and also discovers [`mission.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/mission.md) files in lesson directories, compiling them into the `ARTIFACTS` array.

## Generating the Final data.js File

With `phases`, `glossaryTerms`, and `artifacts` collected, the script constructs the final output at lines 69-78. It writes a JavaScript module to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) containing three exported constants:

```javascript
// Auto-generated by build.js — do not edit manually.
const PHASES   = [...];   // full curriculum tree
const GLOSSARY = [...];   // glossary entries
const ARTIFACTS = [...];  // reusable outputs

```

The file is prefixed with a comment indicating it is auto-generated, preventing manual edits that would be overwritten by subsequent builds.

## Auxiliary Build Tasks

Beyond generating [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js), the `build()` function (lines 19-86) orchestrates several maintenance tasks:

- **Synchronizing README badges** via `syncReadme()` to update completion counts
- **Updating marketing numbers** via `syncCounts()`
- **Generating SEO files** including [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml) via `writeSitemap()` and [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) for AI consumption
- **Recording build metadata** by storing the Git ref in [`build-meta.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/build-meta.js)

These tasks ensure the repository documentation and auxiliary files remain consistent with the actual lesson content.

## Running the Build Locally

To execute the site build process manually or test changes before committing:

```bash

# From the repository root

node site/build.js

```

This command prints progress logs and regenerates [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). You can verify the output by inspecting the generated file:

```bash

# Show the first 20 lines of the generated module

head -n 20 site/data.js

```

To consume the generated data in a site page:

```javascript
import { PHASES, GLOSSARY, ARTIFACTS } from './site/data.js';

// List all completed lessons
const completed = PHASES.flatMap(p => p.lessons)
                       .filter(l => l.status === 'complete')
                       .map(l => l.name);
console.log('Completed lessons:', completed);

```

## Summary

- **The build script** [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) serves as the central pipeline that transforms markdown documentation into structured JavaScript data.
- **Source files** [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) provide the raw content, while the `phases/` directory contains reusable artifacts.
- **Parsing functions** `parseRoadmap()`, `parseReadme()`, and `extractLessonMeta()` extract status, structure, and metadata respectively.
- **Output generation** creates [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) exporting `PHASES`, `GLOSSARY`, and `ARTIFACTS` constants for the website to consume.
- **Automation** via GitHub Actions ensures the site stays synchronized with repository changes without manual data entry.

## Frequently Asked Questions

### What triggers the site build process for generating data.js?

The build executes automatically via GitHub Actions workflows when changes are pushed to the repository, or manually by running `node site/build.js` from the repository root. This ensures [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) always reflects the current state of [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md).

### How does the build determine a lesson's completion status?

The `parseRoadmap()` function checks for emoji indicators in [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) (✅ for complete, 🚧 for in-progress, ⬚ for planned). Additionally, if a lesson row in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) contains a URL, the build forces the status to `complete` regardless of the roadmap indicator.

### Why is data.js auto-generated instead of maintained manually?

Auto-generation ensures a **single source of truth** where [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) remain the authoritative curriculum documents. Manual maintenance would risk divergence between the human-readable documentation and the website data, whereas the build script guarantees consistency.

### What data structures does the generated data.js export?

The file exports three constants: `PHASES` (the curriculum tree with lesson metadata), `GLOSSARY` (term definitions from [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md)), and `ARTIFACTS` (reusable skills, prompts, and agents from the `phases/` directory).