# How site/build.js Generates site/data.js from README and ROADMAP: The Build Pipeline Explained

> Discover how site/build.js generates site/data.js by parsing README and ROADMAP. Learn about the build pipeline for ai-engineering-from-scratch.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-09-01

---

**The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script transforms human-readable curriculum documentation into a structured JavaScript module by parsing [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), enriching the data with URLs and SEO metadata, and serializing the results to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) for the static site.**

The rohitg00/ai-engineering-from-scratch repository maintains its curriculum as markdown documentation, using [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) as the central orchestration tool that generates the website's data layer. This Node.js script ensures the single source of truth remains in the readable documentation files while producing the structured data required by the frontend.

## Loading the Curriculum Source Files

At lines 15-19 of [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js), the script resolves absolute paths for the repository root and defines constants for the three primary documentation sources:

- `REPO_ROOT`: The repository base directory
- `README_PATH`: The master curriculum table ([`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md))
- `ROADMAP_PATH`: The completion status tracker ([`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md))
- `GLOSSARY_PATH`: The searchable terms and definitions ([`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md))

## Parsing the ROADMAP Status Map

The `parseRoadmap()` function (lines 101-133) processes [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to extract completion states. It iterates line-by-line to identify phase headings marked with status emojis—**✅** (complete), **🚧** (in progress), or **⬚** (planned)—and captures the status of each lesson row.

The function returns a nested map structure:

```javascript
{
  "Phase 1": {
    phaseStatus: "complete",
    lessons: {
      "Lesson Title": "complete",
      "Another Lesson": "in-progress"
    }
  }
}

```

## Extracting Lesson Metadata from README

The `parseReadme()` function (core logic at lines 138-286) handles the heavy lifting of parsing the master table in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md). It performs two critical tasks:

**Phase Header Detection**: The script recognizes phase headers across multiple markup variants, establishing the current phase context for subsequent lesson rows.

**Lesson Row Processing** (lines 208-286): For each table row, the script extracts:
- **Lesson name** and optional link
- **Type** classification (Build or Learn)
- **Language** identifier (plain text or emoji)
- **URL construction** using `GITHUB_BASE` when links are present

Crucially, the function cross-references the roadmap map generated earlier, attaching the appropriate status to each lesson and defaulting to **"planned"** when no match exists in the roadmap data.

## Enriching Data with Learning Paths and SEO

Beyond the core curriculum, the pipeline gathers supplemental information and search metadata.

**Learning Paths**: The `parseLearningPaths()` function (lines 307-424) reads JSON files from the `learning-paths/` directory to create custom ordered overlays on the standard curriculum structure.

**SEO Manifest Generation**: The `buildSeoManifests()` function (lines 190-247) creates search-optimized metadata for each lesson. It reads individual lesson documentation (such as [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md)), extracts summaries and keywords, and generates SEO-friendly titles and descriptions. The validation logic (lines 291-423) enforces strict technical SEO limits:
- **Title** ≤ 60 characters
- **Description** ≤ 160 characters
- **Excerpt** ≤ 220 words

The function also validates canonical URL uniqueness to prevent duplicate content issues.

## Serializing the Output Artifacts

The final stage writes multiple artifacts to disk. The `writeFigureManifest()` function (lines 624-633) generates [`figure-manifest.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/figure-manifest.js) for custom diagram components, while `writeSeoArtifacts()` (lines 818-829) produces [`lesson-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lesson-seo.json) and [`certification-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/certification-seo.json).

Finally, the script constructs the main output file. At approximately lines 1240-1255, it calls `fs.writeFileSync(OUTPUT_PATH, ...)` where `OUTPUT_PATH` is defined at line 19 as [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). This writes a JavaScript module exporting the complete curriculum object:

```javascript
export default {
  phases: [...],
  lessons: [...],
  glossary: [...],
  seoManifest: {...}
};

```

## Regenerating the Site Data

To execute the build pipeline locally, run the script from the repository root:

```bash
node site/build.js

```

The script outputs diagnostic logs indicating the files written:

```

wrote figure-manifest.js (123 routed figures)
wrote lesson-seo.json (340 lessons)
wrote certification-seo.json (12 tracks)

```

Consuming the generated module in the frontend requires a standard ES module import:

```html
<script type="module">
  import data from './data.js';
  console.log(data.phases);        // Array of phase objects
  console.log(data.lessonManifest); // SEO-ready lesson metadata
</script>

```

## Summary

- [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) operates as the single-source build script for the rohitg00/ai-engineering-from-scratch curriculum site, processing markdown into structured data.
- The **parsing phase** extracts structured data from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) (lesson metadata) and [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) (completion status with ✅/🚧/⬚ emojis).
- **Cross-referencing** occurs during `parseReadme()`, where roadmap status values are mapped to individual lessons, defaulting to "planned" when unspecified.
- **Validation logic** in `buildSeoManifests()` enforces technical SEO constraints including character limits for titles (≤60) and descriptions (≤160).
- The script generates multiple artifacts: [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) (main data module), [`lesson-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lesson-seo.json), [`certification-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/certification-seo.json), and [`figure-manifest.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/figure-manifest.js).
- Running `node site/build.js` regenerates the entire data layer from markdown source files, maintaining synchronization between documentation and the deployed site.

## Frequently Asked Questions

### What happens if a lesson exists in README.md but not in ROADMAP.md?

The `parseReadme()` function defaults the lesson status to **"planned"** when no matching entry exists in the roadmap map. This ensures the site displays all documented curriculum items even if their implementation status hasn't been explicitly tracked in the roadmap file.

### How does the script handle different markdown formatting for phase headers?

The `parseReadme()` function at lines 138-210 implements flexible header detection that recognizes multiple markup variants for phase declarations. This robust parsing accommodates inconsistent formatting across the long-form markdown document while correctly grouping lessons under their respective phases.

### Where does the SEO validation logic enforce character limits?

The `buildSeoManifests()` function contains validation checks at lines 291-423 that verify SEO metadata adheres to technical constraints: titles must not exceed 60 characters, descriptions must stay within 160 characters, and excerpts are limited to 220 words. The script throws descriptive errors if content violates these limits.

### Can this build process be automated in CI/CD pipelines?

Yes, [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) is a standard Node.js script with no interactive dependencies. You can execute `node site/build.js` within any CI/CD environment that provides Node.js, allowing automated regeneration of [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) and SEO artifacts whenever [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) or [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) changes are merged into the main branch.