# How site/data.js Is Generated from Lesson Markdown Files in AI Engineering from Scratch

> Learn how site/data.js generates from lesson markdown files in AI Engineering from Scratch. Discover the automated process of extracting metadata and content.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-13

---

**The [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) file is auto-generated by [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) by parsing README.md, ROADMAP.md, and individual lesson markdown files to extract metadata, summaries, keywords, glossary terms, and reusable artifacts.**

In the `rohitg00/ai-engineering-from-scratch` repository, the curriculum data powering the lesson viewer and AI-agent maps is not maintained manually. Instead, [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) is programmatically generated from the repository's markdown source files, ensuring the website always reflects the current state of the course materials. This automated pipeline transforms human-readable documentation into a structured JavaScript payload consumed by the front-end.

## The Build Pipeline Architecture

The generation logic lives entirely within **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)**. This Node.js script orchestrates the transformation of the markdown curriculum into structured data. It aggregates information from multiple sources across the repository to construct three primary exports: `PHASES`, `GLOSSARY`, and `ARTIFACTS`.

The script performs six distinct operations: parsing curriculum metadata, extracting roadmap status, processing individual lesson documentation, building the glossary index, discovering reusable artifacts, and writing the final output file.

## Parsing Curriculum Metadata from README.md

The script first reads **[`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md)** to identify the lesson structure. In the "Parse README.md" section (lines 63-135), the parser scans for tables containing "# | Lesson | Type | Lang" columns. Each row represents a lesson entry that gets added to the curriculum index, establishing the foundational structure of the course.

## Extracting Status from ROADMAP.md

Next, the script consults **[`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md)** to determine completion status. The `parseRoadmap` function (lines 31-60) maps each phase and lesson to its current state—whether complete, in-progress, or planned—using the status indicators (✅/🚧/⬚) found in the roadmap tables.

## Processing Individual Lesson Documentation

For every lesson with a valid GitHub link, the script resolves the relative path and loads its **[`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md)** file. The `extractLessonMeta` function (lines 33-68) performs two critical extractions:

- The first blockquote becomes the **summary** field
- All `###` headings are collected as **keywords**

These fields attach to the lesson object as `lesson.summary` and `lesson.keywords`, providing searchable metadata for each lesson.

## Building the Glossary Index

The `parseGlossary` function (lines 71-103) processes **[`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md)** to build the `GLOSSARY` array. This provides a centralized reference for terminology used throughout the curriculum, making definitions available to the front-end application.

## Discovering Reusable Artifacts

The `discoverArtifacts` function (lines 40-97) traverses each lesson's **`outputs/`** folder to locate reusable components. It identifies **skills**, **prompts**, **agents**, and **missions** stored as markdown files under `phases/*/*/outputs/*.md`. These artifacts represent the deliverables and reusable assets produced by each lesson.

## Assembling and Writing the Output

After collecting all data, the script assembles the final structures:

- **`PHASES`**: Array of phase objects containing lessons with metadata, status, URLs, summaries, and keywords
- **`GLOSSARY`**: Parsed glossary terms
- **`ARTIFACTS`**: Discovered output artifacts

The script then writes [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) using `JSON.stringify` (lines 49-58), prepending a header comment indicating the file is auto-generated with a build timestamp. The file exports these three constants for consumption by the lesson viewer and AI-agent mapping tools.

## Local Regeneration Workflow

While the CI pipeline triggers this automatically on every push, you can regenerate the data file locally:

```bash
cd /path/to/ai-engineering-from-scratch
node site/build.js

```

This prints progress to stdout and creates a fresh [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) file ready for the front-end.

## Consuming the Generated Data

The output file exports three constants that power the site's functionality:

```javascript
// src/lesson-index.js
import { PHASES, GLOSSARY, ARTIFACTS } from '../site/data.js';

export function getLessonByName(name) {
  for (const phase of PHASES) {
    const lesson = phase.lessons.find(l => l.name === name);
    if (lesson) return lesson;
  }
  return null;
}

// Filter completed lessons
export const completedLessons = PHASES.flatMap(p =>
  p.lessons.filter(l => l.status === 'complete')
);

```

Inspecting individual lesson metadata:

```javascript
import { PHASES } from '../site/data.js';

const lesson = PHASES[0].lessons[0];
console.log('Summary:', lesson.summary);
console.log('Keywords:', lesson.keywords);

```

## Summary

- **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** is the single source of truth for data generation, processing multiple markdown sources into structured JavaScript
- **[`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md)** drives the curriculum structure through parsed lesson tables
- **[`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md)** provides completion status via the `parseRoadmap` function (lines 31-60)
- Individual **[`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md)** files supply summaries and keywords through `extractLessonMeta` (lines 33-68)
- **[`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md)** and **`phases/*/*/outputs/`** folders provide glossary terms and reusable artifacts
- Output is written to **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** as exportable constants (`PHASES`, `GLOSSARY`, `ARTIFACTS`) using `JSON.stringify`

## Frequently Asked Questions

### What triggers the regeneration of site/data.js?

The file regenerates automatically during CI pipeline execution on every push to the repository. You can also trigger it manually by running `node site/build.js` from the repository root, which parses all markdown sources and writes a fresh data file with a current timestamp.

### Which markdown files are parsed to generate the curriculum data?

The parser reads [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) for lesson listings, [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for status tracking, [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md) for terminology, and each lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file for content metadata. Additionally, it scans `phases/*/*/outputs/*.md` for reusable artifacts.

### How does the script extract lesson summaries and keywords?

The `extractLessonMeta` function (lines 33-68) extracts the first blockquote from each lesson's markdown as the summary and collects all H3 headings as an array of keywords. These populate the `lesson.summary` and `lesson.keywords` fields in the generated data.

### Where are reusable artifacts like prompts and agents discovered?

The `discoverArtifacts` function (lines 40-97) scans the `phases/*/*/outputs/` directories for markdown files, categorizing them as skills, prompts, agents, or missions based on their location and content. These are aggregated into the `ARTIFACTS` export.