# How the AI Engineering from Scratch Curriculum Teaches Computer Vision: From CNNs to Vision Transformers

> Master computer vision from scratch. Our AI engineering curriculum builds understanding from CNNs to Vision Transformers, progressing historically and constructing concepts bottom-up.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-18

---

**The curriculum follows a historical-to-modern progression, teaching computer vision through bottom-up construction—from manual 2D convolutions to Vision Transformers—where each architectural innovation builds upon the previous layer of understanding.**

The `rohitg00/ai-engineering-from-scratch` repository structures its computer vision curriculum as an evolutionary narrative that transforms how learners understand visual AI. Rather than treating models as isolated black boxes, the curriculum's approach to computer vision reveals how each breakthrough (from ReLU activations to residual connections to patch embeddings) historically responded to the limitations of prior architectures. This methodology grounds students in `phases/04-computer-vision/`, where they implement every layer from scratch before encountering abstraction libraries.

## The Evolutionary Stack: From Pixels to Transformers

The curriculum organizes computer vision into six progressive stages that mirror the field's actual development. Students begin with **image fundamentals** in [`phases/04-computer-vision/01-image-fundamentals/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/01-image-fundamentals/docs/en.md), learning to treat pixels as tensors, then graduate to **manual 2D convolutions** in [`phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md) to understand the "conv" primitive without framework magic.

The core CNN sequence spans [`phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md), where learners implement successive architectural families—LeNet-5, AlexNet, VGG, Inception, and ResNet—each introducing exactly one novel concept (ReLU, depth, 3×3 stacks, parallel filters, residual skips). This leads to **advanced vision tasks** including YOLO for object detection and Mask-RCNN for segmentation, culminating in **Vision Transformers (ViT)** at [`phases/04-computer-vision/14-vision-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/14-vision-transformers/docs/en.md) and modern hybrids like Swin and ConvNeXt.

## CNN Foundations: Building Classic Architectures

### LeNet-5: The Convolutional Template

The curriculum introduces the first convolutional template through LeNet-5, implemented in [`phases/04-computer-vision/03-cnns-lenet-to-resnet/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/03-cnns-lenet-to-resnet/code/main.py). This establishes the foundational pattern: convolution → activation → pooling → fully-connected layers.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class LeNet5(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
        self.pool = nn.AvgPool2d(2)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, num_classes)

    def forward(self, x):
        x = self.pool(torch.tanh(self.conv1(x)))
        x = self.pool(torch.tanh(self.conv2(x)))
        x = torch.flatten(x, 1)
        x = torch.tanh(self.fc1(x))
        x = torch.tanh(self.fc2(x))
        return self.fc3(x)

```

### VGG: The 3×3 Recipe

Moving to the VGG family, the curriculum demonstrates how stacking only 3×3 convolutions with padding achieves deeper receptive fields while maintaining parameter efficiency. The `VGGBlock` class in the same file shows the batch normalization and ReLU activation pattern that became standard.

```python
class VGGBlock(nn.Module):
    def __init__(self, in_c, out_c):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, kernel_size=3, padding=1)
        self.bn1   = nn.BatchNorm2d(out_c)
        self.conv2 = nn.Conv2d(out_c, out_c, kernel_size=3, padding=1)
        self.bn2   = nn.BatchNorm2d(out_c)
        self.pool  = nn.MaxPool2d(2)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        return self.pool(x)

```

### ResNet: The Skip Connection Revolution

The curriculum treats ResNet as the pivotal bridge to modern architectures, focusing on the **BasicBlock** implementation that solves the degradation problem through residual shortcuts. Students learn that the `self.shortcut` path—whether `Identity` or a 1×1 convolution—enables gradients to flow directly through deep networks.

```python
class BasicBlock(nn.Module):
    def __init__(self, in_c, out_c, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, 3, stride, 1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_c)
        self.conv2 = nn.Conv2d(out_c, out_c, 3, 1, 1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_c)

        self.shortcut = nn.Identity()
        if stride != 1 or in_c != out_c:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_c, out_c, 1, stride, bias=False),
                nn.BatchNorm2d(out_c)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = out + self.shortcut(x)
        return F.relu(out)

```

## Vision Transformers: Repurposing NLP for Computer Vision

The transition to Vision Transformers in [`phases/04-computer-vision/14-vision-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/14-vision-transformers/docs/en.md) teaches students to treat images as sequences of patches. The curriculum emphasizes that **patch embedding** is mathematically equivalent to a convolution with stride equal to kernel size, explicitly connecting the CNN primitives to the transformer paradigm.

The implementation in [`phases/04-computer-vision/14-vision-transformers/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/14-vision-transformers/code/main.py) demonstrates three critical components: the `PatchEmbedding` layer that spatially rearranges image tensors, pre-layer normalization (pre-LN) residuals inherited from NLP, and the learned positional embeddings that replace convolutional inductive biases.

```python
import torch, torch.nn as nn

class PatchEmbedding(nn.Module):
    def __init__(self, in_ch=3, patch=16, dim=192, img_sz=64):
        super().__init__()
        assert img_sz % patch == 0
        self.proj = nn.Conv2d(in_ch, dim, patch, patch)
        self.n_patch = (img_sz // patch) ** 2

    def forward(self, x):
        x = self.proj(x)                     # (B, dim, H/patch, W/patch)

        return x.flatten(2).transpose(1, 2) # (B, N, dim)

class Block(nn.Module):
    def __init__(self, dim, heads, mlp_ratio=4):
        super().__init__()
        self.ln1 = nn.LayerNorm(dim)
        self.att = nn.MultiheadAttention(dim, heads, batch_first=True)
        self.ln2 = nn.LayerNorm(dim)
        self.mlp = nn.Sequential(
            nn.Linear(dim, dim * mlp_ratio),
            nn.GELU(),
            nn.Linear(dim * mlp_ratio, dim)
        )

    def forward(self, x):
        x = x + self.att(self.ln1(x), self.ln1(x), self.ln1(x), need_weights=False)[0]
        x = x + self.mlp(self.ln2(x))
        return x

class ViT(nn.Module):
    def __init__(self, img_sz=64, patch=16, dim=192, depth=6, heads=3, num_classes=10):
        super().__init__()
        self.patch = PatchEmbedding(3, patch, dim, img_sz)
        self.cls = nn.Parameter(torch.zeros(1, 1, dim))
        self.pos = nn.Parameter(torch.zeros(1, self.patch.n_patch + 1, dim))
        self.blocks = nn.ModuleList([Block(dim, heads) for _ in range(depth)])
        self.ln = nn.LayerNorm(dim)
        self.head = nn.Linear(dim, num_classes)

    def forward(self, x):
        x = self.patch(x)
        cls = self.cls.expand(x.shape[0], -1, -1)
        x = torch.cat([cls, x], dim=1) + self.pos
        for b in self.blocks:
            x = b(x)
        return self.head(self.ln(x[:, 0]))

```

## Modern Hybrid Architectures: Swin and ConvNeXt

The curriculum concludes by examining **Swin Transformers** (windowed attention) and **ConvNeXt** (modernized CNNs) to demonstrate architectural convergence. According to the source documents in the ViT lesson, the only fundamental difference between a ConvNeXt block and a ViT block is the **locality bias**—ConvNeXt uses 7×7 depthwise convolutions while ViT uses global self-attention. This section teaches that training recipe differences (optimization, augmentation, regularization) often dominate performance gaps rather than architectural choices alone.

## Pedagogical Design Principles

The computer vision curriculum employs five specific instructional strategies that differentiate it from traditional deep learning courses:

1. **Bottom-up construction** – Every lesson starts with the lowest-level operation (pixel tensor → manual convolution) and adds exactly one new concept at a time, preventing cognitive overload.

2. **Progressive code reuse** – The same `torch.nn.Conv2d` primitive introduced in the convolutions lesson reappears as the patch embedding projection in ViT, reinforcing the mathematical equivalence between operations.

3. **Backbone-agnostic mindset** – After covering CNN families, students learn to treat any model (ResNet, Swin, ConvNeXt, ViT) as a **backbone** producing feature maps, then attach task-specific heads (classifiers, detectors, segmenters). This mirrors production pipelines where backbone selection is a hyperparameter.

4. **Explicit "what changed?" documentation** – Each lesson identifies the single architectural novelty differentiating families (e.g., *skip connections* in ResNet, *multi-branch* in Inception, *patch embedding* in ViT), making the evolutionary thread explicit.

5. **Compact implementations** – All lessons provide PyTorch implementations under 40 lines, allowing students to run models, inspect tensor shapes, and count parameters before progressing to abstraction libraries.

## Summary

- The curriculum structures computer vision education as a historical progression from LeNet-5 to Vision Transformers, with each lesson in `phases/04-computer-vision/` building upon the previous architectural primitive.
- Students implement convolutional layers manually before using `nn.Conv2d`, then observe how the same convolutional primitives become patch embeddings in ViT.
- **Residual connections** serve as the conceptual bridge between deep CNNs and transformer blocks, appearing in both ResNet's `BasicBlock` and the pre-LN residual paths of Vision Transformers.
- The backbone-agnostic mindset treats ResNet, Swin, ConvNeXt, and ViT as interchangeable feature extractors, preparing learners for real-world computer vision engineering where model selection is task-dependent.
- All implementations remain under 40 lines of PyTorch code, emphasizing understanding over abstraction.

## Frequently Asked Questions

### Why does the curriculum teach CNNs before Vision Transformers?

The curriculum follows the historical development of computer vision because each architectural limitation motivated the next breakthrough. Understanding why ResNet needed skip connections (degradation in deep networks) explains why Vision Transformers adopted pre-layer normalization residuals. This progression ensures students grasp *why* transformers discard convolutional priors rather than treating them as arbitrary alternatives.

### How does the curriculum explain the relationship between ResNet and Vision Transformers?

According to the source documents in [`phases/04-computer-vision/14-vision-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/14-vision-transformers/docs/en.md), the curriculum explicitly teaches that **residual connections**—the innovation that made deep CNNs trainable in ResNet—became the backbone of every modern transformer block. The `BasicBlock` implementation's `out + self.shortcut(x)` pattern reappears in the ViT `Block` class as `x + self.att(...)` and `x + self.mlp(...)`, demonstrating architectural continuity.

### What is the "backbone-agnostic" mindset taught in the computer vision curriculum?

The curriculum trains students to view any encoder—whether the `LeNet5` convolutional stack or the `ViT` transformer blocks—as a **backbone** that produces a feature representation. After the CNN families section, all subsequent lessons (object detection, segmentation, depth estimation) demonstrate how to attach task-specific heads to any backbone, treating the feature extractor as a configurable hyperparameter rather than a fixed architecture.

### How are the code implementations structured in the computer vision phases?

Every lesson in `phases/04-computer-vision/` provides a compact PyTorch implementation under 40 lines, located in [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py) files. For example, the Vision Transformer implementation includes explicit `PatchEmbedding`, `Block`, and `ViT` classes that students can run, inspect with `print(model)`, and modify. This hands-on approach ensures learners understand tensor shapes and parameter counts before encountering high-level abstractions.