How site/data.js Is Generated from Lesson Markdown Files in AI Engineering from Scratch

The site/data.js file is auto-generated by site/build.js by parsing README.md, ROADMAP.md, and individual lesson markdown files to extract metadata, summaries, keywords, glossary terms, and reusable artifacts.

In the rohitg00/ai-engineering-from-scratch repository, the curriculum data powering the lesson viewer and AI-agent maps is not maintained manually. Instead, site/data.js is programmatically generated from the repository's markdown source files, ensuring the website always reflects the current state of the course materials. This automated pipeline transforms human-readable documentation into a structured JavaScript payload consumed by the front-end.

The Build Pipeline Architecture

The generation logic lives entirely within site/build.js. This Node.js script orchestrates the transformation of the markdown curriculum into structured data. It aggregates information from multiple sources across the repository to construct three primary exports: PHASES, GLOSSARY, and ARTIFACTS.

The script performs six distinct operations: parsing curriculum metadata, extracting roadmap status, processing individual lesson documentation, building the glossary index, discovering reusable artifacts, and writing the final output file.

Parsing Curriculum Metadata from README.md

The script first reads README.md to identify the lesson structure. In the "Parse README.md" section (lines 63-135), the parser scans for tables containing "# | Lesson | Type | Lang" columns. Each row represents a lesson entry that gets added to the curriculum index, establishing the foundational structure of the course.

Extracting Status from ROADMAP.md

Next, the script consults ROADMAP.md to determine completion status. The parseRoadmap function (lines 31-60) maps each phase and lesson to its current state—whether complete, in-progress, or planned—using the status indicators (✅/🚧/⬚) found in the roadmap tables.

Processing Individual Lesson Documentation

For every lesson with a valid GitHub link, the script resolves the relative path and loads its docs/en.md file. The extractLessonMeta function (lines 33-68) performs two critical extractions:

• The first blockquote becomes the summary field
• All ### headings are collected as keywords

These fields attach to the lesson object as lesson.summary and lesson.keywords, providing searchable metadata for each lesson.

Building the Glossary Index

The parseGlossary function (lines 71-103) processes glossary/terms.md to build the GLOSSARY array. This provides a centralized reference for terminology used throughout the curriculum, making definitions available to the front-end application.

Discovering Reusable Artifacts

The discoverArtifacts function (lines 40-97) traverses each lesson's outputs/ folder to locate reusable components. It identifies skills, prompts, agents, and missions stored as markdown files under phases/*/*/outputs/*.md. These artifacts represent the deliverables and reusable assets produced by each lesson.

Assembling and Writing the Output

After collecting all data, the script assembles the final structures:

• PHASES: Array of phase objects containing lessons with metadata, status, URLs, summaries, and keywords
• GLOSSARY: Parsed glossary terms
• ARTIFACTS: Discovered output artifacts

The script then writes site/data.js using JSON.stringify (lines 49-58), prepending a header comment indicating the file is auto-generated with a build timestamp. The file exports these three constants for consumption by the lesson viewer and AI-agent mapping tools.

Local Regeneration Workflow

While the CI pipeline triggers this automatically on every push, you can regenerate the data file locally:

cd /path/to/ai-engineering-from-scratch
node site/build.js

This prints progress to stdout and creates a fresh site/data.js file ready for the front-end.

Consuming the Generated Data

The output file exports three constants that power the site's functionality:

// src/lesson-index.js
import { PHASES, GLOSSARY, ARTIFACTS } from '../site/data.js';

export function getLessonByName(name) {
  for (const phase of PHASES) {
    const lesson = phase.lessons.find(l => l.name === name);
    if (lesson) return lesson;
  }
  return null;
}

// Filter completed lessons
export const completedLessons = PHASES.flatMap(p =>
  p.lessons.filter(l => l.status === 'complete')
);

Inspecting individual lesson metadata:

import { PHASES } from '../site/data.js';

const lesson = PHASES[0].lessons[0];
console.log('Summary:', lesson.summary);
console.log('Keywords:', lesson.keywords);

Summary

• site/build.js is the single source of truth for data generation, processing multiple markdown sources into structured JavaScript
• README.md drives the curriculum structure through parsed lesson tables
• ROADMAP.md provides completion status via the parseRoadmap function (lines 31-60)
• Individual docs/en.md files supply summaries and keywords through extractLessonMeta (lines 33-68)
• glossary/terms.md and phases/*/*/outputs/ folders provide glossary terms and reusable artifacts
• Output is written to site/data.js as exportable constants (PHASES, GLOSSARY, ARTIFACTS) using JSON.stringify

Frequently Asked Questions

What triggers the regeneration of site/data.js?

The file regenerates automatically during CI pipeline execution on every push to the repository. You can also trigger it manually by running node site/build.js from the repository root, which parses all markdown sources and writes a fresh data file with a current timestamp.

Which markdown files are parsed to generate the curriculum data?

The parser reads README.md for lesson listings, ROADMAP.md for status tracking, glossary/terms.md for terminology, and each lesson's docs/en.md file for content metadata. Additionally, it scans phases/*/*/outputs/*.md for reusable artifacts.

How does the script extract lesson summaries and keywords?

The extractLessonMeta function (lines 33-68) extracts the first blockquote from each lesson's markdown as the summary and collects all H3 headings as an array of keywords. These populate the lesson.summary and lesson.keywords fields in the generated data.

Where are reusable artifacts like prompts and agents discovered?

The discoverArtifacts function (lines 40-97) scans the phases/*/*/outputs/ directories for markdown files, categorizing them as skills, prompts, agents, or missions based on their location and content. These are aggregated into the ARTIFACTS export.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}