What Is Covered in the Computer Vision Phase of the AI Engineering From Scratch Curriculum

Phase 4 of the AI Engineering From Scratch curriculum comprises 28 incremental lessons that advance learners from raw pixel tensors and classic CNNs to Vision Transformers, self-supervised pre-training, 3D depth estimation, and state-of-the-art video diffusion models.

The Computer Vision phase in the rohitg00/ai-engineering-from-scratch repository is a hands-on curriculum designed to bridge theory and production engineering. Documented in phases/04-computer-vision/README.md, this phase employs a rigorous Build-It / Use-It methodology where every lesson includes conceptual explanations, self-contained Python implementations, and deterministic validation tests.

Six Logical Blocks Spanning 28 Lessons

The curriculum is organized into six progressive blocks, each targeting distinct competencies in modern computer vision.

Foundations: Pixels, Convolutions, and Classic CNNs (Lessons 01-04)

This block establishes the numerical foundation of vision systems. Learners work with raw pixel tensors, implement basic convolutions, and construct classic architectures from LeNet through ResNet. The 01-image-fundamentals/docs/en.md file introduces the canonical preprocessing pipeline, teaching the critical distinction between HWC (height-width-channel) and CHW layouts required by PyTorch. The block culminates in transfer learning strategies for image classification.

Segmentation and Object Detection (Lessons 05-08)

Moving beyond classification, these lessons cover dense prediction tasks. Learners implement U-Net architectures for semantic segmentation (documented in 07-semantic-segmentation-unet/docs/en.md), configure YOLO pipelines for real-time object detection, and explore instance segmentation with Mask R-CNN. Advanced transfer-learning tricks for fine-tuning on limited data are emphasized throughout.

Vision Transformers and Generative Models (Lessons 09-14)

This section transitions from convolutional inductive biases to attention mechanisms. The curriculum covers GAN-based image generation, foundational diffusion models, and the complete implementation of Vision Transformers (ViT). In 14-vision-transformers/docs/en.md, learners construct patch embeddings and positional encodings from scratch, understanding how transformer architectures process non-sequential image data.

Self-Supervised Learning and Vision-Language Models (Lessons 15-18)

Focusing on representation learning without labels, these lessons implement contrastive methods including SimCLR, DINO, and MAE (Masked Autoencoders). The 17-self-supervised-vision/docs/en.md module also introduces open-vocabulary retrieval using CLIP and the construction of multimodal vision-language model (VLM) pipelines.

Depth Estimation and 3D Vision (Lessons 19-22)

Expanding into spatial understanding, this block covers monocular depth estimation, geometric transforms, and 3D representation. Learners implement OCR and document understanding systems, generate neural radiance fields (NeRF), and experiment with 3D Gaussian splatting point-cloud pipelines as detailed in 26-monocular-depth/docs/en.md.

Video Understanding and Advanced Diffusion (Lessons 23-28)

The final block addresses temporal dynamics and generative video. Lessons cover Diffusion Transformers and Rectified Flow, world-model video diffusion architectures, multi-object tracking algorithms, and advanced video diffusion techniques. The capstone content in 28-world-models-video-diffusion/docs/en.md synthesizes preceding concepts into generative video systems.

The Build-It / Use-It Pedagogical Pattern

Every lesson in the Computer Vision phase follows a consistent four-step validation pattern:

1. Explain – A concise conceptual write-up establishes theoretical grounding (e.g., "An image is a tensor of light samples" in the fundamentals module).
2. Build – A self-contained implementation resides in code/main.py, typically in Python or occasionally Rust.
3. Ship – Generated artifacts include model-ready tensors, structured prompts, or skill markdown files suitable for production pipelines.
4. Validate – Unit tests in the tests/ directory enforce determinism and correctness, ensuring implementations match theoretical specifications.

Core Implementation: Tensor Preprocessing and Normalization

The Image Fundamentals lesson establishes the preprocessing pipeline reused across all 28 lessons. The following excerpt from 01-image-fundamentals/docs/en.md demonstrates converting synthetic RGB data into a standardized model input:

import numpy as np
import torch
import torch.nn.functional as F

# 1️⃣ Create a deterministic synthetic RGB image (H, W, C)

def synthetic_rgb(h=128, w=192, seed=0):
    rng = np.random.default_rng(seed)
    yy, xx = np.meshgrid(np.linspace(0, 1, h), np.linspace(0, 1, w), indexing="ij")
    r = (np.sin(xx * 6) * 0.5 + 0.5) * 255
    g = yy * 255
    b = (1 - yy) * xx * 255
    rgb = np.stack([r, g, b], axis=-1) + rng.normal(0, 6, (h, w, 3))
    return np.clip(rgb, 0, 255).astype(np.uint8)

img_hwc = synthetic_rgb()

# 2️⃣ Convert HWC → CHW for PyTorch

img_chw = img_hwc.transpose(2, 0, 1)               # NumPy

tensor = torch.from_numpy(img_chw).float() / 255.0 # Normalize to [0,1]

# 3️⃣ Apply ImageNet mean/std standardization

mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
std  = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)
tensor = (tensor.unsqueeze(0) - mean) / std         # Shape: (1, C, H, W)

print(f"Tensor dtype: {tensor.dtype}")
print(f"Shape (N, C, H, W): {tensor.shape}")
print(f"Per‑channel mean ≈ {tensor.mean(dim=(0,2,3)).tolist()}")
print(f"Per‑channel std  ≈ {tensor.std(dim=(0,2,3)).tolist()}")

This pattern—synthetic generation, layout transposition, and ImageNet standardization—supports every downstream task in the phase, from segmentation mask generation to video frame sampling.

Summary

• The Computer Vision phase consists of 28 lessons divided into six blocks: Foundations, Segmentation/Detection, Vision Transformers, Self-Supervision/VLMs, Depth/3D, and Video/Diffusion.
• Each lesson follows the Build-It / Use-It pattern with Explain, Build, Ship, and Validate stages.
• Core skills progress from pixel tensor manipulation and CNN arithmetic to Vision Transformers, self-supervised learning, 3D Gaussian splatting, and video diffusion models.
• All implementations are validated by unit tests and follow canonical preprocessing standards established in 01-image-fundamentals/docs/en.md.

Frequently Asked Questions

How many lessons are included in the Computer Vision phase?

The Computer Vision phase contains 28 lessons numbered 01 through 28. These are grouped into six logical blocks that progress from basic image classification to advanced video diffusion and world models.

What programming languages are used in Phase 4 implementations?

The primary implementation language is Python, utilizing PyTorch for deep learning operations. Occasional lessons include Rust implementations for performance-critical components, particularly in low-level tensor operations or optimized inference pipelines.

Does the curriculum cover both 2D and 3D computer vision?

Yes. Lessons 01-18 focus on 2D vision including classification, segmentation, object detection, and image generation. Lessons 19-22 transition to 3D vision with monocular depth estimation, NeRF, and 3D Gaussian splatting. Lessons 23-28 address temporal and video understanding.

Are transformer architectures covered in the Computer Vision phase?

Absolutely. Vision Transformers (ViT) are covered in lessons 09-14, where learners implement patch embeddings and positional encodings from scratch. Later lessons introduce Diffusion Transformers and multimodal transformers for vision-language tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}