# What Is Covered in the Computer Vision Phase of the AI Engineering From Scratch Curriculum

> Explore Phase 4 of AI Engineering From Scratch. Master computer vision from CNNs to Vision Transformers, self-supervised learning, depth estimation, and video diffusion models.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-08-30

---

**Phase 4 of the AI Engineering From Scratch curriculum comprises 28 incremental lessons that advance learners from raw pixel tensors and classic CNNs to Vision Transformers, self-supervised pre-training, 3D depth estimation, and state-of-the-art video diffusion models.**

The **Computer Vision phase** in the `rohitg00/ai-engineering-from-scratch` repository is a hands-on curriculum designed to bridge theory and production engineering. Documented in [`phases/04-computer-vision/README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/README.md), this phase employs a rigorous **Build-It / Use-It** methodology where every lesson includes conceptual explanations, self-contained Python implementations, and deterministic validation tests.

## Six Logical Blocks Spanning 28 Lessons

The curriculum is organized into six progressive blocks, each targeting distinct competencies in modern computer vision.

### Foundations: Pixels, Convolutions, and Classic CNNs (Lessons 01-04)

This block establishes the numerical foundation of vision systems. Learners work with raw pixel tensors, implement basic convolutions, and construct classic architectures from LeNet through ResNet. The [`01-image-fundamentals/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/01-image-fundamentals/docs/en.md) file introduces the canonical preprocessing pipeline, teaching the critical distinction between **HWC** (height-width-channel) and **CHW** layouts required by PyTorch. The block culminates in transfer learning strategies for image classification.

### Segmentation and Object Detection (Lessons 05-08)

Moving beyond classification, these lessons cover dense prediction tasks. Learners implement **U-Net** architectures for semantic segmentation (documented in [`07-semantic-segmentation-unet/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/07-semantic-segmentation-unet/docs/en.md)), configure **YOLO** pipelines for real-time object detection, and explore instance segmentation with **Mask R-CNN**. Advanced transfer-learning tricks for fine-tuning on limited data are emphasized throughout.

### Vision Transformers and Generative Models (Lessons 09-14)

This section transitions from convolutional inductive biases to attention mechanisms. The curriculum covers **GAN-based image generation**, foundational diffusion models, and the complete implementation of **Vision Transformers (ViT)**. In [`14-vision-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/14-vision-transformers/docs/en.md), learners construct patch embeddings and positional encodings from scratch, understanding how transformer architectures process non-sequential image data.

### Self-Supervised Learning and Vision-Language Models (Lessons 15-18)

Focusing on representation learning without labels, these lessons implement contrastive methods including **SimCLR**, **DINO**, and **MAE** (Masked Autoencoders). The [`17-self-supervised-vision/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/17-self-supervised-vision/docs/en.md) module also introduces open-vocabulary retrieval using **CLIP** and the construction of multimodal **vision-language model (VLM)** pipelines.

### Depth Estimation and 3D Vision (Lessons 19-22)

Expanding into spatial understanding, this block covers monocular depth estimation, geometric transforms, and 3D representation. Learners implement **OCR** and document understanding systems, generate neural radiance fields (**NeRF**), and experiment with **3D Gaussian splatting** point-cloud pipelines as detailed in [`26-monocular-depth/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/26-monocular-depth/docs/en.md).

### Video Understanding and Advanced Diffusion (Lessons 23-28)

The final block addresses temporal dynamics and generative video. Lessons cover **Diffusion Transformers** and Rectified Flow, world-model video diffusion architectures, multi-object tracking algorithms, and advanced video diffusion techniques. The capstone content in [`28-world-models-video-diffusion/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/28-world-models-video-diffusion/docs/en.md) synthesizes preceding concepts into generative video systems.

## The Build-It / Use-It Pedagogical Pattern

Every lesson in the Computer Vision phase follows a consistent four-step validation pattern:

1. **Explain** – A concise conceptual write-up establishes theoretical grounding (e.g., "An image is a tensor of light samples" in the fundamentals module).
2. **Build** – A self-contained implementation resides in [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py), typically in Python or occasionally Rust.
3. **Ship** – Generated artifacts include model-ready tensors, structured prompts, or skill markdown files suitable for production pipelines.
4. **Validate** – Unit tests in the `tests/` directory enforce determinism and correctness, ensuring implementations match theoretical specifications.

## Core Implementation: Tensor Preprocessing and Normalization

The Image Fundamentals lesson establishes the preprocessing pipeline reused across all 28 lessons. The following excerpt from [`01-image-fundamentals/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/01-image-fundamentals/docs/en.md) demonstrates converting synthetic RGB data into a standardized model input:

```python
import numpy as np
import torch
import torch.nn.functional as F

# 1️⃣ Create a deterministic synthetic RGB image (H, W, C)

def synthetic_rgb(h=128, w=192, seed=0):
    rng = np.random.default_rng(seed)
    yy, xx = np.meshgrid(np.linspace(0, 1, h), np.linspace(0, 1, w), indexing="ij")
    r = (np.sin(xx * 6) * 0.5 + 0.5) * 255
    g = yy * 255
    b = (1 - yy) * xx * 255
    rgb = np.stack([r, g, b], axis=-1) + rng.normal(0, 6, (h, w, 3))
    return np.clip(rgb, 0, 255).astype(np.uint8)

img_hwc = synthetic_rgb()

# 2️⃣ Convert HWC → CHW for PyTorch

img_chw = img_hwc.transpose(2, 0, 1)               # NumPy

tensor = torch.from_numpy(img_chw).float() / 255.0 # Normalize to [0,1]

# 3️⃣ Apply ImageNet mean/std standardization

mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std  = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
tensor = (tensor.unsqueeze(0) - mean) / std         # Shape: (1, C, H, W)

print(f"Tensor dtype: {tensor.dtype}")
print(f"Shape (N, C, H, W): {tensor.shape}")
print(f"Per‑channel mean ≈ {tensor.mean(dim=(0,2,3)).tolist()}")
print(f"Per‑channel std  ≈ {tensor.std(dim=(0,2,3)).tolist()}")

```

This pattern—synthetic generation, layout transposition, and ImageNet standardization—supports every downstream task in the phase, from segmentation mask generation to video frame sampling.

## Summary

- The **Computer Vision phase** consists of **28 lessons** divided into six blocks: Foundations, Segmentation/Detection, Vision Transformers, Self-Supervision/VLMs, Depth/3D, and Video/Diffusion.
- Each lesson follows the **Build-It / Use-It** pattern with Explain, Build, Ship, and Validate stages.
- Core skills progress from **pixel tensor manipulation** and **CNN arithmetic** to **Vision Transformers**, **self-supervised learning**, **3D Gaussian splatting**, and **video diffusion models**.
- All implementations are validated by unit tests and follow canonical preprocessing standards established in [`01-image-fundamentals/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/01-image-fundamentals/docs/en.md).

## Frequently Asked Questions

### How many lessons are included in the Computer Vision phase?

The Computer Vision phase contains **28 lessons** numbered 01 through 28. These are grouped into six logical blocks that progress from basic image classification to advanced video diffusion and world models.

### What programming languages are used in Phase 4 implementations?

The primary implementation language is **Python**, utilizing PyTorch for deep learning operations. Occasional lessons include **Rust** implementations for performance-critical components, particularly in low-level tensor operations or optimized inference pipelines.

### Does the curriculum cover both 2D and 3D computer vision?

Yes. Lessons 01-18 focus on **2D vision** including classification, segmentation, object detection, and image generation. Lessons 19-22 transition to **3D vision** with monocular depth estimation, NeRF, and 3D Gaussian splatting. Lessons 23-28 address **temporal and video understanding**.

### Are transformer architectures covered in the Computer Vision phase?

Absolutely. **Vision Transformers (ViT)** are covered in lessons 09-14, where learners implement patch embeddings and positional encodings from scratch. Later lessons introduce **Diffusion Transformers** and multimodal transformers for vision-language tasks.