# How to Add a New Lesson Following AGENTS.md Conventions in AI Engineering From Scratch

> Learn how to add a new lesson following AGENTS.md conventions in AI Engineering From Scratch. Scaffold, document, and commit your lesson to the repository.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-08-31

---

**To add a new lesson following AGENTS.md conventions, scaffold the lesson directory using [`scripts/scaffold-lesson.sh`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/scaffold-lesson.sh), populate the required documentation and code files according to the front-matter schema, and commit the changes atomically with the message format `feat(phase-NN/MM): <Lesson Title>`.**

Adding new content to the **ai-engineering-from-scratch** curriculum requires strict adherence to the architectural standards defined in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md). The repository enforces a specific folder structure, documentation schema, and validation pipeline to ensure all 435 lessons remain coherent and compatible with the automated site builder and certification engine.

## Three-Stage Workflow for Adding Lessons

The contribution workflow mirrors the process outlined in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md) and consists of three distinct stages: scaffolding, content population, and public indexing.

### Stage 1: Scaffold the Lesson Skeleton

Begin by using the official scaffolding script to generate the mandatory folder hierarchy. Run [`scripts/scaffold-lesson.sh`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/scaffold-lesson.sh) with the phase identifier, lesson slug, and title:

```bash
scripts/scaffold-lesson.sh 05-nlp-foundations-to-advanced 03-tokenizers "Tokenizers from Scratch"

```

This script performs several operations defined in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md): it creates the directory `phases/05-nlp-foundations-to-advanced/03-tokenizers/` with subfolders `docs/`, `code/`, `tests/`, and `outputs/`; prepopulates [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) with the front-matter template; generates a stub [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py); and prints the next steps for editing.

### Stage 2: Populate Lesson Content

Follow the front-matter schema strictly when editing [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md). The metadata must include `title`, `hook`, `Type`, `Languages`, `Prerequisites`, and `Time` according to the specification.

Implement the reference solution in `code/main.<lang>` using only approved standard library dependencies. Create at least **5 unit tests** in `code/tests/` to validate the implementation. Add a [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) file conforming to the 6-question format: 1 pre-assessment question, 3 checkpoint questions, and 2 post-assessment questions.

If the lesson produces reusable artifacts such as prompts, skills, agents, or MCP servers, place these under the `outputs/` directory.

### Stage 3: Wire Into the Public View

Update the public lesson table by inserting a markdown link row into [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) under the appropriate phase section. Use the format:

```markdown
| 03 | [Tokenizers from Scratch](phases/05-nlp-foundations-to-advanced/03-tokenizers) | Build | Python |

```

Modify [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) to reflect the lesson's current status. Ensure all changes are committed as a single atomic unit with the conventional commit message pattern `feat(phase-NN/MM): <Lesson Title>`.

## Manual Scaffolding Alternative

If you prefer not to use the automation script, create the structure manually:

```bash
mkdir -p phases/05-nlp-foundations-to-advanced/03-tokenizers/{code,tests,docs,outputs}
cp LESSON_TEMPLATE.md phases/05-nlp-foundations-to-advanced/03-tokenizers/docs/en.md

```

Then add a minimal implementation stub in [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py):

```python
def main():
    """Implement the tokenization algorithm from scratch."""
    # TODO: add code here

    raise NotImplementedError

if __name__ == "__main__":
    main()

```

Create the required [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) with the exact 6-question schema:

```json
{
  "lesson": "03-tokenizers",
  "title": "Tokenizers from Scratch",
  "questions": [
    {"stage":"pre","question":"What is the purpose of a tokenizer?","options":["Split text","Encode images","Generate embeddings","None"],"correct":0,"explanation":"Tokenizers break raw text into discrete units."},
    {"stage":"check","question":"Which token type preserves whitespace?","options":["WordPiece","Byte‑Pair","Whitespace‑preserving","SentencePiece"],"correct":2,"explanation":"Whitespace-preserving tokenizers maintain spacing information."},
    {"stage":"check","question":"Example checkpoint question?","options":["a","b","c","d"],"correct":1,"explanation":"Detailed rationale here."},
    {"stage":"check","question":"Third checkpoint question?","options":["a","b","c","d"],"correct":2,"explanation":"Explanation of correct answer."},
    {"stage":"post","question":"First post-assessment question?","options":["a","b","c","d"],"correct":3,"explanation":"Final review concept."},
    {"stage":"post","question":"Second post-assessment question?","options":["a","b","c","d"],"correct":0,"explanation":"Summary evaluation."}
  ]
}

```

## Validation and Quality Assurance

Before submitting, run the local validation suite to ensure compliance. Execute [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) to perform schema checks and test discovery, and run [`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py) to verify that [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) statistics match the actual lesson count.

The CI pipeline will automatically regenerate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) and [`catalog.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/catalog.json) from your source files. Never commit these generated artifacts; the build system creates them fresh during deployment.

## Key Architectural Constraints

- **One-lesson-per-commit**: Each lesson must occupy exactly one commit to guarantee clear history and simplify review processes.
- **Front-matter enforcement**: The site builder ([`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)) and certification engine require strict adherence to the metadata schema for lesson discoverability.
- **Standard library first**: The dependency allowlist restricts imports to approved libraries for the lesson's specific language, ensuring reproducibility across environments.

## Summary

- Use [`scripts/scaffold-lesson.sh`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/scaffold-lesson.sh) to generate the `docs/`, `code/`, `tests/`, and `outputs/` folder structure automatically.
- Populate [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) with the required front-matter schema and implement at least 5 unit tests in `code/tests/`.
- Create a 6-question [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) following the 1-pre/3-check/2-post format.
- Wire the lesson into [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) and update [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) before committing with the conventional message `feat(phase-NN/MM): <Lesson Title>`.
- Validate locally with [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) and [`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py) before pushing.

## Frequently Asked Questions

### What is the exact folder structure required for a new lesson?

The repository mandates a four-directory layout inside `phases/<phase-name>/<lesson-slug>/`: `docs/` for markdown documentation, `code/` for implementation files, `tests/` for unit tests, and `outputs/` for reusable artifacts. The [`scaffold-lesson.sh`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scaffold-lesson.sh) script creates this hierarchy automatically according to [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md) specifications.

### How many tests are required for each lesson?

Every lesson must include at least **5 unit tests** placed in the `code/tests/` directory. These tests validate the reference implementation and ensure the educational objectives are technically sound.

### Can I use external libraries in my lesson code?

No, the curriculum follows a strict standard-library-first policy defined in the dependency allowlist. Only approved libraries for the lesson's specific language may be imported. This constraint ensures that learners focus on fundamental algorithms without framework abstraction layers.

### Why are my changes to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) not persisting?

Files like [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) and [`catalog.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/catalog.json) are generated automatically by the CI pipeline based on your lesson's front-matter and should never be committed manually. The build system regenerates these assets during deployment to keep the website and certification tracks synchronized.