# How Website Data Is Generated from README and ROADMAP in AI Engineering from Scratch

> Learn how website data is generated from README and ROADMAP files using a Node.js script in the ai-engineering-from-scratch repository. Understand metadata extraction and dependency validation.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-09-02

---

**The website data is generated by a deterministic Node.js build script that parses [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), extracts phase and lesson metadata, validates curriculum dependencies, and outputs a consolidated [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js) module consumed by the frontend.**

The **AI Engineering from Scratch** repository by `rohitg00` uses a static site generator approach to transform human-authored Markdown documentation into structured website data. Rather than manually maintaining separate data files, the project relies on a single Node.js script located at [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) to automatically derive lesson statuses, curriculum prerequisites, and phase metadata directly from the repository's root documentation files.

## The Build Script Architecture

At the core of the data generation pipeline is **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)**, a Node.js script orchestrated by the top-level `build()` function. This script executes automatically on every Vercel deployment and can be triggered locally to regenerate the site's data layer. The architecture follows a functional parsing pattern where distinct extractor functions handle specific markdown structures, ensuring a single source of truth for curriculum content.

The script performs three primary parsing operations: **roadmap status extraction** (tracking completion states), **README catalogue parsing** (extracting lesson metadata), and **curriculum graph validation** (parsing the Mermaid dependency diagram). These discrete operations merge into a unified data structure that powers the entire frontend.

## Step-by-Step Data Transformation Pipeline

The generation process follows a deterministic seven-step pipeline that transforms raw Markdown into typed JavaScript modules.

### Loading Source Files

The build process begins by loading the canonical source documents from the repository root. According to the source code in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) (lines 19-27), the script resolves absolute paths to both documentation files and loads them as UTF-8 strings:

```javascript
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const readme = fs.readFileSync(README_PATH, 'utf8');
const roadmap = fs.readFileSync(ROADMAP_PATH, 'utf8');

```

These files serve as the single source of truth for all curriculum data displayed on the website.

### Parsing the Roadmap Status Matrix

The **`parseRoadmap`** function (lines 104-133 of [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)) processes **ROADMAP.md** to extract the status matrix. This function scans the Markdown table to identify lesson completion states, creating a nested map structure of phases → lessons → status values (`complete`, `in-progress`, `planned`).

The parser specifically extracts table rows formatted as `| 01 | Dev Environment | ✅ |`, converting emoji indicators into normalized status strings. This status map later attaches to lesson objects during the merge phase, ensuring the website displays real-time progress indicators without manual JSON updates.

### Extracting the Phase and Lesson Catalogue

Next, **`parseReadme`** (lines 138-190) walks through **README.md** to construct the lesson catalogue. This function identifies Phase headers (e.g., "Phase 1: Foundations") and parses the lesson tables that follow each header. For every lesson discovered, the parser records:

- **Name** and **type** classification
- **Language badges** (normalizing emoji indicators like 🐍 into plain language names)
- **GitHub URL** (`lesson.url`)
- **Implicit status** (marking lessons as 'complete' if they contain links)

This extraction ensures that lesson metadata remains synchronized with the repository's primary documentation.

### Deriving the Curriculum Dependency Graph

The **`parseCurriculumPrereqs`** function (lines 360-405) handles the Mermaid diagram embedded in README.md labeled "The shape of the curriculum." This parser validates dependency edges between phases, enforces a single root node (Phase 0), and constructs prerequisite lists for each learning stage.

By parsing the visual curriculum graph into structured data, the build script enables the website to render interactive learning paths and validation warnings when prerequisites are unmet.

### Merging Sources and Generating Static Data

In the final transformation stage, the script merges roadmap-derived statuses with the README lesson catalogue. This enrichment process produces a canonical list of phases and lessons with accurate status fields. The script then serializes these structures into **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** (lines 820-886), utilizing a template literal that exports frozen data constants:

```javascript
const OUTPUT_PATH = path.join(__dirname, 'data.js');
const output = `// Auto-generated …
const ROADMAP_PREREQS = ${JSON.stringify(roadmapPrereqs, null, 2)};
const PHASES = ${JSON.stringify(phases, null, 2)};
…
`;
fs.writeFileSync(OUTPUT_PATH, output, 'utf8');

```

Additionally, the script generates **[`site/lesson-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/lesson-seo.json)** and **[`site/certification-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/certification-seo.json)** for search engine optimization metadata.

## Frontend Integration and Generated Artifacts

Beyond the core [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js) module, the build script refreshes multiple UI artifacts to ensure consistency across the site. The **`syncReadme`** function updates badge counts within [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) itself, while **`syncCounts`** regenerates static HTML pages. The script also injects discovery tables into **[`site/catalog.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/catalog.html)** and **[`site/certifications.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/certifications.html)**, writes a sitemap via **`writeSitemap`**, and generates **[`site/llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/llms.txt)**—a human-readable curriculum summary optimized for AI agents and external indexing.

The frontend consumes the generated data through standard ES6 imports:

```javascript
import { PHASES, ROADMAP_PREREQS } from './data.js';

// Example: list all completed lessons
const completed = PHASES.flatMap(p =>
  p.lessons.filter(l => l.status === 'complete')
);
console.log(`✅ ${completed.length} lessons are finished`);

```

## Running the Build Locally

Developers can regenerate the website data locally without triggering a Vercel deployment. Execute the build script from the repository root to update all generated files:

```bash

# From the repo root

node site/build.js      # prints progress and generates site/data.js

```

This command processes the current state of [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), validates the curriculum graph, and writes updated artifacts to the `site/` directory.

## Summary

- **Single script architecture**: The entire data pipeline runs through [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js), ensuring deterministic transformations from Markdown to JavaScript modules.
- **Dual-source parsing**: The script extracts lesson metadata from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and completion statuses from [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), merging them into a unified curriculum structure.
- **Graph validation**: The Mermaid dependency diagram in README.md is parsed and validated to ensure Phase 0 serves as the single root and all prerequisite edges are logical.
- **Multi-format output**: Beyond [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js), the build generates SEO manifests, HTML catalog injections, sitemaps, and AI-readable summaries ([`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt)).
- **Automated synchronization**: The build updates badge counts and discovery tables automatically, eliminating manual synchronization between documentation and website data.

## Frequently Asked Questions

### What triggers the website data generation?

The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script executes automatically during every Vercel deployment. For local development, running `node site/build.js` manually triggers the same deterministic transformation pipeline that parses [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to regenerate the static data files.

### How does the build script determine lesson completion status?

The script uses a two-tier status resolution system. First, **`parseRoadmap`** extracts explicit status indicators (✅, 🚧, or ⬚) from the table in [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md). Second, **`parseReadme`** infers completion by detecting the presence of GitHub URLs in lesson rows. These sources merge during the enrichment phase, with the roadmap values taking precedence for explicit state management.

### What is the curriculum dependency graph?

The curriculum dependency graph is a Mermaid diagram embedded within [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) that visualizes learning path prerequisites across phases. The **`parseCurriculumPrereqs`** function parses this diagram into structured prerequisite lists, validating that Phase 0 serves as the single root node and that no circular dependencies exist between learning stages.

### Can I run the build process locally?

Yes. Execute `node site/build.js` from the repository root to process the current Markdown files and regenerate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) along with all SEO manifests and HTML artifacts. This local execution mirrors the Vercel deployment process, allowing developers to preview data changes before committing.