How the AI Engineering from Scratch Curriculum Teaches Computer Vision: From CNNs to Vision Transformers

The curriculum follows a historical-to-modern progression, teaching computer vision through bottom-up construction—from manual 2D convolutions to Vision Transformers—where each architectural innovation builds upon the previous layer of understanding.

The rohitg00/ai-engineering-from-scratch repository structures its computer vision curriculum as an evolutionary narrative that transforms how learners understand visual AI. Rather than treating models as isolated black boxes, the curriculum's approach to computer vision reveals how each breakthrough (from ReLU activations to residual connections to patch embeddings) historically responded to the limitations of prior architectures. This methodology grounds students in phases/04-computer-vision/, where they implement every layer from scratch before encountering abstraction libraries.

The Evolutionary Stack: From Pixels to Transformers

The curriculum organizes computer vision into six progressive stages that mirror the field's actual development. Students begin with image fundamentals in phases/04-computer-vision/01-image-fundamentals/docs/en.md, learning to treat pixels as tensors, then graduate to manual 2D convolutions in phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md to understand the "conv" primitive without framework magic.

The core CNN sequence spans phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md, where learners implement successive architectural families—LeNet-5, AlexNet, VGG, Inception, and ResNet—each introducing exactly one novel concept (ReLU, depth, 3×3 stacks, parallel filters, residual skips). This leads to advanced vision tasks including YOLO for object detection and Mask-RCNN for segmentation, culminating in Vision Transformers (ViT) at phases/04-computer-vision/14-vision-transformers/docs/en.md and modern hybrids like Swin and ConvNeXt.

CNN Foundations: Building Classic Architectures

LeNet-5: The Convolutional Template

The curriculum introduces the first convolutional template through LeNet-5, implemented in phases/04-computer-vision/03-cnns-lenet-to-resnet/code/main.py. This establishes the foundational pattern: convolution → activation → pooling → fully-connected layers.

import torch, torch.nn as nn, torch.nn.functional as F

class LeNet5(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
        self.pool = nn.AvgPool2d(2)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, num_classes)

    def forward(self, x):
        x = self.pool(torch.tanh(self.conv1(x)))
        x = self.pool(torch.tanh(self.conv2(x)))
        x = torch.flatten(x, 1)
        x = torch.tanh(self.fc1(x))
        x = torch.tanh(self.fc2(x))
        return self.fc3(x)

VGG: The 3×3 Recipe

Moving to the VGG family, the curriculum demonstrates how stacking only 3×3 convolutions with padding achieves deeper receptive fields while maintaining parameter efficiency. The VGGBlock class in the same file shows the batch normalization and ReLU activation pattern that became standard.

class VGGBlock(nn.Module):
    def __init__(self, in_c, out_c):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, kernel_size=3, padding=1)
        self.bn1   = nn.BatchNorm2d(out_c)
        self.conv2 = nn.Conv2d(out_c, out_c, kernel_size=3, padding=1)
        self.bn2   = nn.BatchNorm2d(out_c)
        self.pool  = nn.MaxPool2d(2)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        return self.pool(x)

ResNet: The Skip Connection Revolution

The curriculum treats ResNet as the pivotal bridge to modern architectures, focusing on the BasicBlock implementation that solves the degradation problem through residual shortcuts. Students learn that the self.shortcut path—whether Identity or a 1×1 convolution—enables gradients to flow directly through deep networks.

class BasicBlock(nn.Module):
    def __init__(self, in_c, out_c, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, 3, stride, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_c)
        self.conv2 = nn.Conv2d(out_c, out_c, 3, 1, 1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_c)

        self.shortcut = nn.Identity()
        if stride != 1 or in_c != out_c:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_c, out_c, 1, stride, bias=False),
                nn.BatchNorm2d(out_c)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + self.shortcut(x)
        return F.relu(out)

Vision Transformers: Repurposing NLP for Computer Vision

The transition to Vision Transformers in phases/04-computer-vision/14-vision-transformers/docs/en.md teaches students to treat images as sequences of patches. The curriculum emphasizes that patch embedding is mathematically equivalent to a convolution with stride equal to kernel size, explicitly connecting the CNN primitives to the transformer paradigm.

The implementation in phases/04-computer-vision/14-vision-transformers/code/main.py demonstrates three critical components: the PatchEmbedding layer that spatially rearranges image tensors, pre-layer normalization (pre-LN) residuals inherited from NLP, and the learned positional embeddings that replace convolutional inductive biases.

import torch, torch.nn as nn

class PatchEmbedding(nn.Module):
    def __init__(self, in_ch=3, patch=16, dim=192, img_sz=64):
        super().__init__()
        assert img_sz % patch == 0
        self.proj = nn.Conv2d(in_ch, dim, patch, patch)
        self.n_patch = (img_sz // patch) ** 2

    def forward(self, x):
        x = self.proj(x)                     # (B, dim, H/patch, W/patch)

        return x.flatten(2).transpose(1, 2) # (B, N, dim)

class Block(nn.Module):
    def __init__(self, dim, heads, mlp_ratio=4):
        super().__init__()
        self.ln1 = nn.LayerNorm(dim)
        self.att = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.ln2 = nn.LayerNorm(dim)
        self.mlp = nn.Sequential(
            nn.Linear(dim, dim * mlp_ratio),
            nn.GELU(),
            nn.Linear(dim * mlp_ratio, dim)
        )

    def forward(self, x):
        x = x + self.att(self.ln1(x), self.ln1(x), self.ln1(x), need_weights=False)[0]
        x = x + self.mlp(self.ln2(x))
        return x

class ViT(nn.Module):
    def __init__(self, img_sz=64, patch=16, dim=192, depth=6, heads=3, num_classes=10):
        super().__init__()
        self.patch = PatchEmbedding(3, patch, dim, img_sz)
        self.cls = nn.Parameter(torch.zeros(1, 1, dim))
        self.pos = nn.Parameter(torch.zeros(1, self.patch.n_patch + 1, dim))
        self.blocks = nn.ModuleList([Block(dim, heads) for _ in range(depth)])
        self.ln = nn.LayerNorm(dim)
        self.head = nn.Linear(dim, num_classes)

    def forward(self, x):
        x = self.patch(x)
        cls = self.cls.expand(x.shape[0], -1, -1)
        x = torch.cat([cls, x], dim=1) + self.pos
        for b in self.blocks:
            x = b(x)
        return self.head(self.ln(x[:, 0]))

Modern Hybrid Architectures: Swin and ConvNeXt

The curriculum concludes by examining Swin Transformers (windowed attention) and ConvNeXt (modernized CNNs) to demonstrate architectural convergence. According to the source documents in the ViT lesson, the only fundamental difference between a ConvNeXt block and a ViT block is the locality bias—ConvNeXt uses 7×7 depthwise convolutions while ViT uses global self-attention. This section teaches that training recipe differences (optimization, augmentation, regularization) often dominate performance gaps rather than architectural choices alone.

Pedagogical Design Principles

The computer vision curriculum employs five specific instructional strategies that differentiate it from traditional deep learning courses:

1. Bottom-up construction – Every lesson starts with the lowest-level operation (pixel tensor → manual convolution) and adds exactly one new concept at a time, preventing cognitive overload.

2. Progressive code reuse – The same torch.nn.Conv2d primitive introduced in the convolutions lesson reappears as the patch embedding projection in ViT, reinforcing the mathematical equivalence between operations.

3. Backbone-agnostic mindset – After covering CNN families, students learn to treat any model (ResNet, Swin, ConvNeXt, ViT) as a backbone producing feature maps, then attach task-specific heads (classifiers, detectors, segmenters). This mirrors production pipelines where backbone selection is a hyperparameter.

4. Explicit "what changed?" documentation – Each lesson identifies the single architectural novelty differentiating families (e.g., skip connections in ResNet, multi-branch in Inception, patch embedding in ViT), making the evolutionary thread explicit.

5. Compact implementations – All lessons provide PyTorch implementations under 40 lines, allowing students to run models, inspect tensor shapes, and count parameters before progressing to abstraction libraries.

Summary

• The curriculum structures computer vision education as a historical progression from LeNet-5 to Vision Transformers, with each lesson in phases/04-computer-vision/ building upon the previous architectural primitive.
• Students implement convolutional layers manually before using nn.Conv2d, then observe how the same convolutional primitives become patch embeddings in ViT.
• Residual connections serve as the conceptual bridge between deep CNNs and transformer blocks, appearing in both ResNet's BasicBlock and the pre-LN residual paths of Vision Transformers.
• The backbone-agnostic mindset treats ResNet, Swin, ConvNeXt, and ViT as interchangeable feature extractors, preparing learners for real-world computer vision engineering where model selection is task-dependent.
• All implementations remain under 40 lines of PyTorch code, emphasizing understanding over abstraction.

Frequently Asked Questions

Why does the curriculum teach CNNs before Vision Transformers?

The curriculum follows the historical development of computer vision because each architectural limitation motivated the next breakthrough. Understanding why ResNet needed skip connections (degradation in deep networks) explains why Vision Transformers adopted pre-layer normalization residuals. This progression ensures students grasp why transformers discard convolutional priors rather than treating them as arbitrary alternatives.

How does the curriculum explain the relationship between ResNet and Vision Transformers?

According to the source documents in phases/04-computer-vision/14-vision-transformers/docs/en.md, the curriculum explicitly teaches that residual connections—the innovation that made deep CNNs trainable in ResNet—became the backbone of every modern transformer block. The BasicBlock implementation's out + self.shortcut(x) pattern reappears in the ViT Block class as x + self.att(...) and x + self.mlp(...), demonstrating architectural continuity.

What is the "backbone-agnostic" mindset taught in the computer vision curriculum?

The curriculum trains students to view any encoder—whether the LeNet5 convolutional stack or the ViT transformer blocks—as a backbone that produces a feature representation. After the CNN families section, all subsequent lessons (object detection, segmentation, depth estimation) demonstrate how to attach task-specific heads to any backbone, treating the feature extractor as a configurable hyperparameter rather than a fixed architecture.

How are the code implementations structured in the computer vision phases?

Every lesson in phases/04-computer-vision/ provides a compact PyTorch implementation under 40 lines, located in code/main.py files. For example, the Vision Transformer implementation includes explicit PatchEmbedding, Block, and ViT classes that students can run, inspect with print(model), and modify. This hands-on approach ensures learners understand tensor shapes and parameter counts before encountering high-level abstractions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}