How to Build Neural Networks from Scratch: The Complete AI Engineering Curriculum

The AI Engineering from Scratch repository by rohitg00 is a self‑contained, open‑source curriculum that teaches you to construct every component of modern AI—from perceptrons to autonomous agents—by implementing each algorithm from first principles before touching a framework.

Learning to build neural networks from scratch requires more than copying PyTorch tutorials. The rohitg00/ai-engineering-from-scratch repository provides a staged, 503‑lesson curriculum that forces you to derive weight updates, backpropagation, and attention mechanisms using only NumPy before graduating to production frameworks.

The "Build-It / Use-It" Pedagogy

Every lesson in the curriculum follows a strict Build-It / Use-It split. You first construct the algorithm from raw mathematics—such as implementing the perceptron weight update rule with nothing but NumPy—before executing the same concept via a high‑level library like PyTorch.

In phases/03-deep-learning-core/01-the-perceptron/, the docs/en.md file walks you through the linear algebra of the activation function and gradient descent. Only after you have manually computed the forward and backward passes does the curriculum introduce torch.nn.Linear. This spine guarantees you understand what a framework does under the hood, making you capable of debugging convergence failures and designing custom architectures.

Repository Architecture and Phase Structure

The repository is organized into 20 numbered phases ranging from phases/00-setup-and-tooling to phases/14-agent-engineering. Each phase contains a logical grouping of lessons that progress from fundamentals to advanced topics like multi‑agent orchestration.

Lesson Anatomy

Every lesson lives in its own folder with a strict, predictable layout:

phases/<NN>-<phase-name>/<NN>-<lesson-name>/
├── docs/en.md          # narrative & theory

├── code/               # runnable implementation (Python, TypeScript, Rust, Julia)

└── outputs/            # generated artifact: prompt, skill, agent, or MCP server

This structure enforces reproducibility. The code/ directory contains runnable implementations, while outputs/ houses artifacts that can be dropped into production systems.

Coding Your First Perceptron from Scratch

To begin building neural networks from scratch, navigate to the foundational deep‑learning lesson:

git clone https://github.com/rohitg00/ai-engineering-from-scratch.git
cd ai-engineering-from-scratch
python phases/03-deep-learning-core/01-the-perceptron/code/main.py

The main.py file implements the perceptron training loop using only NumPy, matching the mathematical narrative in the lesson’s documentation. You will initialize weights, compute the dot product of inputs and weights, apply a step function, and manually adjust weights based on the error delta. This raw implementation mirrors the mathematics described in docs/en.md before any abstraction layers are introduced.

From Learning to Deployment: Reusable Artifacts

The curriculum treats education as infrastructure. Each lesson generates concrete artifacts stored in the outputs/ directory—ranging from system prompts to fully functional agents. These are not hypothetical exercises; they are production‑ready components.

To install all generated skills and prompts for immediate reuse with LLM agents:

python3 scripts/install_skills.py

The scripts/install_skills.py utility walks every outputs/ directory, registers each SKILL.md or prompt file, and makes them available to agents such as Claude or OpenAI’s function‑calling interface.

Building MCP Servers for Model Context

Later phases introduce the Model‑Context Protocol (MCP), a language‑agnostic interface for LLMs to exchange context. The lesson in phases/13-tools-and-protocols/07-building-an-mcp-server/ provides a reference implementation:

cd phases/13-tools-and-protocols/07-building-an-mcp-server
python code/server.py

This FastAPI‑based server implements the MCP transport layer, allowing any MCP‑compatible client to query tools and context managed by your custom server. It bridges the gap between isolated scripts and autonomous systems.

Autonomous Agent Loops

Phase 14 introduces agent engineering patterns. The phases/14-agent-engineering/01-the-agent-loop/ directory contains a minimal Python agent loop that demonstrates tool usage, memory management, and planning algorithms. This code serves as the foundation for building multi‑agent systems that can reason over long‑term context.

Automated Curriculum Validation

Maintaining 503 lessons across multiple languages requires rigorous automation. The repository ships with CI jobs defined in .github/workflows/curriculum.yml that audit lesson structure and regenerate documentation.

Key validation scripts include:

• scripts/audit_lessons.py – Enforces the directory structure and validates that every lesson contains the required docs/en.md, code/, and outputs/ directories.
• scripts/check_readme_counts.py – Ensures the root README.md accurately reflects the current lesson and phase counts.
• site/build.js – Generates the static site (site/data.js) used for the online curriculum viewer, ensuring the web documentation stays synchronized with the repository content.

These tools guarantee that as the curriculum evolves, every file path, code snippet, and artifact remains consistent and executable.

Summary

• The curriculum employs a "Build-It / Use-It" methodology, requiring raw NumPy implementations before permitting PyTorch abstractions.
• Lessons follow a strict three‑part structure: narrative documentation, first‑principles code, and reusable outputs/ artifacts.
• Automation via scripts/audit_lessons.py, scripts/check_readme_counts.py, and .github/workflows/curriculum.yml maintains integrity across 503 lessons.
• Graduates progress from manual perceptrons in phases/03-deep-learning-core/ to production MCP servers and autonomous agent loops.

Frequently Asked Questions

Do I need prior machine learning experience to start this curriculum?

No. The phases/00-setup-and-tooling phase assumes only basic Python and linear algebra. Each lesson builds from first principles, making it accessible to beginners while remaining rigorous for experienced engineers who want to understand the mathematics behind backpropagation.

How is building neural networks from scratch different from using PyTorch?

When you implement gradient descent manually in phases/03-deep-learning-core before using torch.nn, you understand gradient flow, numerical stability, and initialization schemes. This depth is essential for debugging vanishing gradients and optimizing production models that standard tutorials treat as black boxes.

Can I use the generated artifacts in commercial projects?

Yes. The outputs/ directory in each lesson contains production‑ready prompts, skills, and MCP servers. Running scripts/install_skills.py registers these components for immediate integration with Claude, OpenAI function calling, or any MCP‑compatible client.

What is the Model Context Protocol (MCP) and why does it matter?

MCP is a standardized, language‑agnostic interface for LLMs to exchange context and tools. The phases/13-tools-and-protocols/07-building-an-mcp-server/code/server.py implementation demonstrates how to expose custom tools to AI agents, enabling your neural networks to interact with external APIs and data sources in a structured, secure manner.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}