# How to Build Neural Networks from Scratch: The Complete AI Engineering Curriculum

> Learn how to build neural networks from scratch with the AI Engineering from Scratch curriculum. Implement every AI component from first principles without frameworks.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-07-19

---

**The AI Engineering from Scratch repository by rohitg00 is a self‑contained, open‑source curriculum that teaches you to construct every component of modern AI—from perceptrons to autonomous agents—by implementing each algorithm from first principles before touching a framework.**

Learning to build neural networks from scratch requires more than copying PyTorch tutorials. The rohitg00/ai-engineering-from-scratch repository provides a staged, 503‑lesson curriculum that forces you to derive weight updates, backpropagation, and attention mechanisms using only NumPy before graduating to production frameworks.

## The "Build-It / Use-It" Pedagogy

Every lesson in the curriculum follows a strict **Build-It / Use-It** split. You first construct the algorithm from raw mathematics—such as implementing the perceptron weight update rule with nothing but NumPy—before executing the same concept via a high‑level library like PyTorch. 

In `phases/03-deep-learning-core/01-the-perceptron/`, the [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file walks you through the linear algebra of the activation function and gradient descent. Only after you have manually computed the forward and backward passes does the curriculum introduce `torch.nn.Linear`. This spine guarantees you understand what a framework does under the hood, making you capable of debugging convergence failures and designing custom architectures.

## Repository Architecture and Phase Structure

The repository is organized into **20 numbered phases** ranging from `phases/00-setup-and-tooling` to `phases/14-agent-engineering`. Each phase contains a logical grouping of lessons that progress from fundamentals to advanced topics like multi‑agent orchestration.

### Lesson Anatomy

Every lesson lives in its own folder with a strict, predictable layout:

```text
phases/<NN>-<phase-name>/<NN>-<lesson-name>/
├── docs/en.md          # narrative & theory

├── code/               # runnable implementation (Python, TypeScript, Rust, Julia)

└── outputs/            # generated artifact: prompt, skill, agent, or MCP server

```

This structure enforces reproducibility. The `code/` directory contains runnable implementations, while `outputs/` houses artifacts that can be dropped into production systems.

## Coding Your First Perceptron from Scratch

To begin building neural networks from scratch, navigate to the foundational deep‑learning lesson:

```bash
git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch
python phases/03-deep-learning-core/01-the-perceptron/code/main.py

```

The [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) file implements the perceptron training loop using only NumPy, matching the mathematical narrative in the lesson’s documentation. You will initialize weights, compute the dot product of inputs and weights, apply a step function, and manually adjust weights based on the error delta. This raw implementation mirrors the mathematics described in [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) before any abstraction layers are introduced.

## From Learning to Deployment: Reusable Artifacts

The curriculum treats education as infrastructure. Each lesson generates concrete artifacts stored in the `outputs/` directory—ranging from system prompts to fully functional agents. These are not hypothetical exercises; they are production‑ready components.

To install all generated skills and prompts for immediate reuse with LLM agents:

```bash
python3 scripts/install_skills.py

```

The [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py) utility walks every `outputs/` directory, registers each [`SKILL.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/SKILL.md) or prompt file, and makes them available to agents such as Claude or OpenAI’s function‑calling interface.

### Building MCP Servers for Model Context

Later phases introduce the **Model‑Context Protocol (MCP)**, a language‑agnostic interface for LLMs to exchange context. The lesson in `phases/13-tools-and-protocols/07-building-an-mcp-server/` provides a reference implementation:

```bash
cd phases/13-tools-and-protocols/07-building-an-mcp-server
python code/server.py

```

This FastAPI‑based server implements the MCP transport layer, allowing any MCP‑compatible client to query tools and context managed by your custom server. It bridges the gap between isolated scripts and autonomous systems.

### Autonomous Agent Loops

Phase 14 introduces agent engineering patterns. The `phases/14-agent-engineering/01-the-agent-loop/` directory contains a minimal Python agent loop that demonstrates tool usage, memory management, and planning algorithms. This code serves as the foundation for building multi‑agent systems that can reason over long‑term context.

## Automated Curriculum Validation

Maintaining 503 lessons across multiple languages requires rigorous automation. The repository ships with CI jobs defined in [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml) that audit lesson structure and regenerate documentation.

Key validation scripts include:

- **[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** – Enforces the directory structure and validates that every lesson contains the required [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md), `code/`, and `outputs/` directories.
- **[`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py)** – Ensures the root [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) accurately reflects the current lesson and phase counts.
- **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** – Generates the static site ([`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)) used for the online curriculum viewer, ensuring the web documentation stays synchronized with the repository content.

These tools guarantee that as the curriculum evolves, every file path, code snippet, and artifact remains consistent and executable.

## Summary

- The curriculum employs a **"Build-It / Use-It"** methodology, requiring raw NumPy implementations before permitting PyTorch abstractions.
- Lessons follow a strict three‑part structure: narrative documentation, first‑principles code, and reusable `outputs/` artifacts.
- Automation via [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py), [`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py), and [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml) maintains integrity across 503 lessons.
- Graduates progress from manual perceptrons in `phases/03-deep-learning-core/` to production MCP servers and autonomous agent loops.

## Frequently Asked Questions

### Do I need prior machine learning experience to start this curriculum?

No. The `phases/00-setup-and-tooling` phase assumes only basic Python and linear algebra. Each lesson builds from first principles, making it accessible to beginners while remaining rigorous for experienced engineers who want to understand the mathematics behind backpropagation.

### How is building neural networks from scratch different from using PyTorch?

When you implement gradient descent manually in `phases/03-deep-learning-core` before using `torch.nn`, you understand gradient flow, numerical stability, and initialization schemes. This depth is essential for debugging vanishing gradients and optimizing production models that standard tutorials treat as black boxes.

### Can I use the generated artifacts in commercial projects?

Yes. The `outputs/` directory in each lesson contains production‑ready prompts, skills, and MCP servers. Running [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py) registers these components for immediate integration with Claude, OpenAI function calling, or any MCP‑compatible client.

### What is the Model Context Protocol (MCP) and why does it matter?

MCP is a standardized, language‑agnostic interface for LLMs to exchange context and tools. The [`phases/13-tools-and-protocols/07-building-an-mcp-server/code/server.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/13-tools-and-protocols/07-building-an-mcp-server/code/server.py) implementation demonstrates how to expose custom tools to AI agents, enabling your neural networks to interact with external APIs and data sources in a structured, secure manner.