# How the AI Engineering from Scratch Static Site is Built: Inside the site/data.js Pipeline

> Discover how the ai-engineering-from-scratch static site is built. Learn about the Node.js build script and the crucial role of site/data.js in the pipeline.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-14

---

**The ai-engineering-from-scratch static site is generated by a Node.js build script ([`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)) that parses Markdown curriculum files and exports three constants—`PHASES`, `GLOSSARY`, and `ARTIFACTS`—into [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), which serves as the runtime data layer for the client-side UI.**

The public website aiengineeringfromscratch.com operates as a pure static site with zero server-side rendering. According to the rohitg00/ai-engineering-from-scratch repository, the build process aggregates curriculum metadata, lesson statuses, and reusable artifacts into a single JavaScript module consumed by the frontend HTML pages.

## Build Architecture and CI Integration

The static site generation is fully automated through GitHub Actions. On every push to the repository, the workflow defined in [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml) triggers the build command:

```bash
node site/build.js

```

This script reads the source Markdown files—including [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), and [`glossary/terms.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/glossary/terms.md)—and writes the generated data to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). Because the generated file is listed in `.gitignore`, the repository stores only the source of truth, while CI produces the runtime artifacts fresh on every deployment.

## The Build Pipeline in site/build.js

The [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script executes an eight-step pipeline to transform raw curriculum Markdown into structured JavaScript data. Each step uses specific helper functions to extract and normalize content.

### Parsing Source Files and Lesson Status

The build begins by reading the primary source files using `fs.readFileSync` (lines 4‑8 in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)). The `parseRoadmap()` function (lines 30‑61) processes [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to detect status emojis—**✅** for complete, **🚧** for in-progress, and **⬚** for planned—and builds a lookup table mapping phases to lesson completion states.

### Extracting Phase and Lesson Metadata

The `parseReadme()` function (lines 63‑131) scans [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) for Markdown tables containing the master curriculum list. It handles both the compact Phase‑0 table and the `<details>` blocks for subsequent phases, extracting lesson names, types, programming languages, and GitHub URLs. For each lesson discovered, `extractLessonMeta()` (lines 33‑48) reads the corresponding [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file to pull the first blockquote (used as a summary) and all `###` headings (used as keywords), enriching the lesson objects with these metadata fields.

### Discovering Reusable Artifacts

The `discoverArtifacts()` function (lines 39‑98) crawls the directory structure under `phases/*/*/outputs/*.md` to locate skill, prompt, and agent definitions. It reads YAML front-matter from each Markdown file (delimited by `---`) to extract `name`, `description`, `tags`, and other metadata, compiling these into the `ARTIFACTS` array. This function also detects optional [`mission.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/mission.md) files to include mission-specific context.

### Assembling the Final Payload

The `build()` function (lines 100‑162) orchestrates the final assembly, combining the phase objects, glossary terms, and artifacts into three arrays. It serializes these into a JavaScript template string and writes the output to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js). Following this, helper functions like `syncCounts()`, `writeSitemap()`, and `writeLlms()` (lines 162‑225) generate auxiliary files including [`stats.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/stats.json) for badge synchronization, [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml) for SEO, and [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) for LLM-friendly indexing.

## The Critical Role of site/data.js

**[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** serves as the bridge between the author-centric Markdown source layout and the viewer-centric static site. The file exports three top-level constants:

```javascript
const PHASES = [...];   // phases → lessons → metadata
const GLOSSARY = [...]; // term definitions
const ARTIFACTS = [...]; // skills / prompts / agents / missions

```

This architecture provides several key advantages:

- **Single Source of Truth**: The frontend never parses Markdown directly. All lesson data, glossary entries, and artifact descriptions are pre-computed, eliminating duplicate parsing logic in the browser.
- **Fast Client-Side Rendering**: As a plain JavaScript module cached like any static asset, [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js) enables instant UI hydration without backend queries.
- **Consistent URL Mapping**: Each lesson receives a numeric `id` derived from directory names, enabling the frontend to construct clean URLs like `/lesson.html?path=phases/03-deep-learning-core/01-the-perceptron`.
- **Extensible Schema**: Adding new content types requires only updating `discoverArtifacts()` in the build script; the frontend automatically receives new data through the existing `ARTIFACTS` export.

## Running the Build Locally

To generate the static data locally for development or debugging:

```bash

# From the repository root

node site/build.js

```

Upon completion, the script outputs [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) alongside status logs:

```

📖 Reading source files...
🔍 Parsing ROADMAP.md...
🔍 Parsing README.md...
✅ Generated site/data.js

```

## Consuming Generated Data in the Browser

The frontend HTML pages import the generated module directly to render dynamic content:

```html
<script type="module">
  import { PHASES, GLOSSARY, ARTIFACTS } from './data.js';

  // Access Phase 3 lessons
  const phase3 = PHASES.find(p => p.id === 3);
  console.log('Phase 3 lessons:', phase3.lessons.map(l => l.name));

  // Query the glossary
  console.log('First term:', GLOSSARY[0]);

  // Filter for reusable prompts
  const prompts = ARTIFACTS.filter(a => a.kind === 'prompt');
  console.log(`Found ${prompts.length} reusable prompts`);
</script>

```

When constructing lesson URLs, the site uses the path extracted from the GitHub URL stored in each lesson object, typically matching the pattern `phases/XX-topic/YY-lesson-name`.

## Summary

- The ai-engineering-from-scratch static site is built by **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)**, a Node.js script that parses [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md), and artifact files to generate **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)**.
- The build pipeline uses functions like **`parseRoadmap()`**, **`parseReadme()`**, and **`discoverArtifacts()`** to extract metadata, lesson statuses, and reusable content.
- **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** exports **`PHASES`**, **`GLOSSARY`**, and **`ARTIFACTS`** as the single runtime data source for the client-side UI.
- The process runs automatically via **[`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml)** on every push, ensuring the site stays synchronized with the Markdown source.
- Generated files including [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js), [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml), and [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) are git-ignored and produced fresh during CI/CD.

## Frequently Asked Questions

### How does the build script determine if a lesson is complete?

The script uses the **`parseRoadmap()`** function (lines 30‑61) to scan [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) for Unicode emojis. A **✅** indicates a complete lesson, **🚧** marks work in progress, and **⬚** denotes planned content. These statuses are mapped to a lookup table and merged with the lesson metadata extracted from [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md).

### Why is site/data.js not checked into the repository?

**[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** is listed in `.gitignore` to prevent stale data from being committed. Because the file is regenerated on every push by the GitHub Actions workflow, storing it in version control would create unnecessary merge conflicts and potentially serve outdated curriculum data to users.

### How are skills, prompts, and agents discovered during the build?

The **`discoverArtifacts()`** function (lines 39‑98) recursively scans `phases/*/*/outputs/*.md` files. It reads the YAML front-matter between `---` delimiters to extract metadata fields like `name`, `description`, and `tags`, then categorizes each file into the `ARTIFACTS` array based on its type and location.

### Can I run the static site build without GitHub Actions?

Yes. Running **`node site/build.js`** from the repository root executes the full pipeline locally using Node.js. This generates [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml), [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt), and other auxiliary files, allowing you to preview changes or debug the curriculum data before pushing to the remote repository.