AI Engineering from Scratch: The Complete 20-Phase Learning Progression

The recommended learning progression follows a linear spine of 20 sequential phases, starting with Python tooling and mathematics in Phase 0–1, progressing through machine learning, deep learning, and transformers in Phases 2–8, mastering LLMs and reinforcement learning in Phases 9–11, and culminating in multimodal AI and autonomous agent engineering in Phases 12–19.

The rohitg00/ai-engineering-from-scratch repository structures AI education as a cumulative journey where each phase builds algorithmic abstractions atop the previous. This AI engineering curriculum learning progression ensures you construct mathematical foundations before implementing neural networks, and master transformer architectures before training production-grade language models. Following the intended order prevents knowledge gaps that typically hinder production engineering skills.

The Linear Spine: 20 Phases Explained

The curriculum is organized as a sequential "spine" where each phase adds a new layer of abstraction and capability. Deviating from this order risks missing the mathematical primitives required for later deep-learning implementations.

Foundation Layers: Phases 0–3

Begin with Phase 0 – Setup & Tooling (phases/00-setup-and-tooling), which establishes a reproducible workstation using Python, Git, Docker, and Jupyter. This prevents hidden dependency conflicts when running later distributed training code.

Proceed to Phase 1 – Math Foundations (phases/01-math-foundations), covering linear algebra, calculus, probability, and optimization. According to the source code, "maths is the floor of the stack; every algorithm later reduces to these operations."

Next, Phase 2 – ML Fundamentals (phases/02-ml-fundamentals) implements classical algorithms—regression, decision trees, SVMs, k-NN, and clustering pipelines. These provide the conceptual backbone for regularization techniques and bias-variance tradeoffs used in modern deep learning.

Finally, Phase 3 – Deep Learning Core (phases/03-deep-learning-core) transitions from perceptrons to multi-layer networks, implementing backpropagation and optimizers from scratch before introducing PyTorch and JAX. Completing this phase ensures you understand the underlying mechanics before relying on high-level frameworks.

Core AI Disciplines: Phases 4–9

With neural network fundamentals established, progress through specialized domains:

• Phase 4 – Computer Vision (phases/04-computer-vision): Implements convolutions, CNNs, detection, segmentation, diffusion models, 3D vision, and Vision Transformers (ViT).

• Phase 5 – NLP Foundations → Advanced (phases/05-nlp-foundations-to-advanced): Covers tokenization, embeddings, seq2seq architectures, and the attention mechanisms that underpin modern LLMs.

• Phase 6 – Speech & Audio (phases/06-speech-and-audio): Processes waveforms and spectrograms, implementing ASR, TTS, and Whisper models using signal-processing primitives shared with vision pipelines.

• Phase 7 – Transformers Deep Dive (phases/07-transformers-deep-dive): Dissects self-attention, multi-head attention, positional encodings, BERT/GPT architectures, Mixture-of-Experts (MoE), and Flash Attention optimizations.

• Phase 8 – Generative AI (phases/08-generative-ai): Explores GANs, VAEs, diffusion models, ControlNet, LoRA fine-tuning, and evaluation metrics.

• Phase 9 – Reinforcement Learning (phases/09-reinforcement-learning): Master MDPs, Q-learning, DQN, policy gradients, PPO, and RLHF—the optimization paradigm essential for fine-tuning LLMs and building autonomous agents.

Large Language Models & Production: Phases 10–11

Phase 10 – LLMs from Scratch (phases/10-llms-from-scratch) represents the synthesis of previous work: building tokenizers, implementing mini-GPT pre-training, managing distributed training, instruction-tuning, RLHF, quantization, and inference optimization.

Phase 11 – LLM Engineering (phases/11-llm-engineering) transitions from model training to production deployment, covering prompt engineering, RAG (Retrieval-Augmented Generation), embedding strategies, caching mechanisms, guardrails, and cost-controlled production applications.

Multimodal Systems & Agents: Phases 12–19

The final stages unify previous disciplines into autonomous systems:

• Phase 12 – Multimodal AI (phases/12-multimodal-ai): Integrates CLIP, BLIP-2, LLaVA, and audio-language models into unified agents capable of cross-modal reasoning.

• Phase 13 – Tools & Protocols (phases/13-tools-and-protocols): Implements the Model-Context-Protocol (MCP) for standardized tool-calling, async execution, security routing, and inter-service communication.

• Phase 14 – Agent Engineering (phases/14-agent-engineering): The capstone phase featuring the minimal agent_loop.py found at phases/14-agent-engineering/01-the-agent-loop/code/agent_loop.py. This pure-Python implementation demonstrates agent loops, memory systems, planning, orchestration, and observability.

• Phases 15–19 – Advanced Topics & Capstone Projects (phases/15-autonomous-systems through phases/19-capstone-projects): Consolidate skills through multi-agent swarms, infrastructure design, ethics, and end-to-end safety gate implementations.

Personalized Entry Points

If you possess existing AI expertise, the curriculum provides a "find-your-level" skill to prevent redundant study. Located at .claude/skills/find-your-level/SKILL.md, this interactive placement quiz asks ten assessment questions and generates a custom path with hour estimates.

Install and run the diagnostic from the repository root:

python scripts/install_skills.py   # installs the skill files

/find-your-level                  # interactive placement quiz

Navigating the Curriculum Structure

The repository organizes learning materials through specific files that govern progression and validation:

• README.md: High-level overview, phase diagram, and getting-started instructions.
• ROADMAP.md: Tracks lesson completion status and phase counts.
• phases/*/README.md: Introduces each phase's specific learning objectives.
• scripts/install_skills.py: Installs reusable prompts and skills generated during lessons.
• scripts/audit_lessons.py: Validates that every lesson follows the curriculum contract.
• site/build.js: Generates the website data (site/data.js) for the online viewer.

Practical Workflow Examples

Execute individual lessons using the structured phase directories. For example, running the linear algebra intuition lesson from Phase 1:

python phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py

Validate your implementations using the unit-test suite provided with each lesson:

cd phases/01-math-foundations/01-linear-algebra-intuition/code
python -m unittest discover tests -v

Install generated artifacts—such as prompts or skills—into your local environment:

python scripts/install_skills.py
cat outputs/prompts/prompt-loss-function-selector.md | your-llm-cli --model gpt-4

For production integration, the curriculum includes Model-Context-Protocol (MCP) client examples:

from mcp_client import MCPClient

client = MCPClient(server_url="http://localhost:8000")
result = client.call_tool("search_web", query="transformer attention math")
print(result)

Summary

• The AI engineering curriculum learning progression follows 20 sequential phases from phases/00-setup-and-tooling to phases/19-capstone-projects, each building mathematical and algorithmic abstractions cumulatively.
• Foundation phases (0–3) establish Python tooling, linear algebra, classical ML, and neural network fundamentals from scratch.
• Core discipline phases (4–9) specialize in computer vision, NLP, audio processing, transformers, generative models, and reinforcement learning.
• Production phases (10–11) cover LLM pre-training, distributed systems, and deployment engineering.
• Capstone phases (12–19) integrate multimodal AI, MCP protocols, and autonomous agent engineering using the agent_loop.py reference implementation.
• Use the find-your-level skill via .claude/skills/find-your-level/SKILL.md to generate personalized entry points if you have prior experience.

Frequently Asked Questions

How long does it take to complete the full AI Engineering from Scratch curriculum?

The repository does not specify fixed durations, but the find-your-level skill generates custom hour estimates based on your diagnostic results. Beginners should expect several months of dedicated study to progress through all 20 phases, while experienced practitioners using the placement quiz may skip early phases and focus on advanced topics like agent engineering or multimodal systems.

Can I skip the mathematics phases if I already know linear algebra and calculus?

While you can use the interactive placement quiz to assess your level, the curriculum strongly recommends verifying foundational knowledge because Phase 3 – Deep Learning Core implements neural networks from scratch using only NumPy. The phases/01-math-foundations directory contains specific implementations that subsequent phases reference directly, making the mathematical notation and code patterns in later lessons dependent on this early exposure.

What is the difference between Phase 10 (LLMs from Scratch) and Phase 11 (LLM Engineering)?

Phase 10 (phases/10-llms-from-scratch) focuses on building and training language models—implementing tokenizers, pre-training mini-GPT architectures, managing distributed training, and applying RLHF. Phase 11 (phases/11-llm-engineering) transitions to production concerns: prompt engineering, RAG implementation, embedding optimization, caching strategies, and guardrails. Phase 10 creates the model; Phase 11 deploys it as a product.

Where can I find the code for the autonomous agent loop mentioned in Phase 14?

The canonical agent loop implementation resides at phases/14-agent-engineering/01-the-agent-loop/code/agent_loop.py. This minimal pure-Python file demonstrates the core agent loop architecture—handling tool calls, memory persistence, planning iterations, and observability hooks—that serves as the cornerstone for the autonomous systems built in Phases 15–19.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}