How Vision Transformers (ViT) Structurally Differ from CNNs in ai-engineering-from-scratch

Vision Transformers process images as sequences of flattened patches using global self-attention, while CNNs employ sliding convolutional kernels that enforce locality and translation equivariance through weight sharing.

Vision Transformers (ViT) and Convolutional Neural Networks (CNNs) solve the same visual recognition problems but rely on fundamentally different architectural primitives. The ai-engineering-from-scratch repository by rohitg00 provides explicit implementations that contrast these approaches, from the patch tokenization logic in phases/07-transformers-deep-dive/09-vision-transformers/code/main.py to the convolutional hierarchies explored in the CNN lessons. Understanding these structural differences helps you choose between CNNs' efficient inductive bias and ViTs' flexible, architecture-agnostic token processing.

Core Primitives: Convolution vs Patch Tokenization

The foundational difference lies in how each architecture ingests spatial data.

CNNs rely on convolution—a local, weight-shared kernel that slides over the image grid, enforcing translation equivariance and capturing hierarchical features through progressively deeper feature maps. This approach is implemented in the repository's CNN lesson covering LeNet to ResNet architectures.

ViT uses patch tokenization—the image is split into non-overlapping patches (typically 16×16), each flattened and linearly projected to a d-model vector dimension. According to the source code in main.py, this is functionally equivalent to a 1×1 convolution with stride equal to the patch size:

# From phases/07-transformers-deep-dive/09-vision-transformers/code/main.py

def patchify(image, patch_size):
    H, W, C = len(image), len(image[0]), len(image[0][0])
    assert H % patch_size == 0 and W % patch_size == 0
    patches = []
    for i in range(0, H, patch_size):
        for j in range(0, W, patch_size):
            patch = []
            for di in range(patch_size):
                for dj in range(patch_size):
                    patch.extend(image[i + di][j + dj])   # flatten P×P×C

            patches.append(patch)
    return patches, (H // patch_size, W // patch_size)

def linear_project(patches, d_model):
    # Simple dense layer (equivalent to a Conv2d with kernel=P, stride=P)

    in_dim = len(patches[0])
    W = [[random.gauss(0, math.sqrt(2/(in_dim+d_model))) for _ in range(d_model)]
         for _ in range(in_dim)]
    return [[sum(x*w for x,w in zip(patch, col)) for col in zip(*W)] for patch in patches]

Spatial Inductive Bias

CNNs bake in spatial assumptions. The convolution kernel's locality provides built-in translation invariance and the theoretical bias that nearby pixels are semantically related. This strong prior allows CNNs to learn efficiently from smaller datasets.

ViT eliminates spatial inductive bias. Without convolution, the model learns spatial relationships exclusively through self-attention mechanisms. To compensate, the repository implementation explicitly adds learnable positional embeddings to give the transformer a notion of patch order, as seen in the cls_and_pos handling (lines 55-66 of the ViT main.py):

# Conceptual representation from the ViT implementation

# CLS token prepended to sequence

cls_token = random_embeds(1, d_model)

# Positional embeddings added to all tokens including CLS

pos_embed = random_embeds(num_patches + 1, d_model)
token_embeddings = patch_embeddings + pos_embed

Processing Hierarchy: Local vs Global

CNNs build deep hierarchies of local feature maps. Early layers detect edges and textures, while deeper layers aggregate these into global object representations through increasing receptive fields. This is evident in the repository's ResNet implementation, where each block processes spatially localized features.

ViT maintains a flat sequence. All patch tokens are processed in parallel by global self-attention, meaning every token can attend to every other token in every layer. Depth is achieved by stacking identical transformer blocks—identical to the architecture used in BERT—rather than through spatial pooling. As documented in phases/07-transformers-deep-dive/09-vision-transformers/docs/en.md, this creates a uniform processing pipeline where spatial relationships are computed dynamically rather than hard-coded through kernel locality.

Weight Sharing and Parameter Scaling

CNNs achieve parameter efficiency through spatial weight sharing—one set of kernel weights slides across all spatial locations. This constraint reduces the parameter count while maintaining strong performance.

ViT scales differently. While each patch uses the same linear projection matrix initially, the subsequent transformer layers lack spatial weight sharing. Each self-attention head computes Query, Key, Value, and Output projections (4 × d² parameters per layer), causing parameter counts to grow quadratically with model dimension. The repository notes that ViT-Base (d=768) contains approximately 86 million parameters compared to ResNet-50's 25 million parameters, as calculated in lines 90-103 of the ViT code.

Output Aggregation Strategies

The final representation extraction also differs structurally:

CNNs typically aggregate features through global average pooling or a final fully-connected layer after the last convolutional block.

ViT prepends a learnable [CLS] token to the input sequence. This special token's final hidden state serves as the global image representation for classification tasks, or the patch embeddings can be used directly for dense prediction tasks. The repository documentation explains this design choice in phases/07-transformers-deep-dive/09-vision-transformers/docs/en.md (lines 38-42).

Practical Implementation Comparison

To illustrate the structural contrast, consider a minimal CNN block versus the ViT patchification:

import torch.nn as nn

# CNN approach: Local, weight-shared convolution

class ConvBlock(nn.Module):
    def __init__(self, in_c, out_c, kernel=3, stride=1):
        super().__init__()
        self.conv = nn.Conv2d(in_c, out_c, kernel_size=kernel, 
                              stride=stride, padding=1)
        self.bn   = nn.BatchNorm2d(out_c)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        return self.relu(self.bn(self.conv(x)))

# ViT approach: Global self-attention on patch tokens

from transformers import ViTImageProcessor, ViTModel

processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTModel.from_pretrained("google/vit-base-patch16-224-in21k")

# Image becomes sequence of 196 patches (14×14) + 1 CLS token

inputs = processor(image, return_tensors="pt")
out = model(**inputs).last_hidden_state  # (batch, 197, 768)

cls_emb = out[:, 0]  # Global image representation

Summary

• CNNs are local, weight-shared, and hierarchical, using convolution kernels that enforce translation equivariance and spatial locality through built-in inductive bias.
• ViT is global, token-based, and architecture-agnostic, converting images to patch sequences processed by standard transformer encoders with learnable positional embeddings.
• Parameter scaling differs significantly: CNNs grow linearly with channels and depth, while ViT scales quadratically with d-model due to self-attention matrices.
• Data requirements vary: CNNs work efficiently with smaller datasets (≤100M images) due to strong priors, while ViT requires massive datasets (ImageNet-21k) or self-supervised pre-training to match performance.
• Implementation location: The core ViT logic resides in phases/07-transformers-deep-dive/09-vision-transformers/code/main.py, while CNN fundamentals are covered in phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md.

Frequently Asked Questions

What is the fundamental structural difference between ViT patch tokenization and CNN convolution?

CNNs use sliding kernels that share weights across spatial locations, enforcing locality and translation equivariance by design. ViT instead splits the image into non-overlapping patches, flattens them, and projects each to a vector embedding—functionally equivalent to a convolution with stride equal to kernel size, but without the spatial weight sharing that characterizes CNNs.

Why do Vision Transformers require more training data than CNNs?

ViT lacks the strong spatial inductive bias built into convolution kernels. While CNNs assume that nearby pixels are related and reuse this pattern across the image, ViT must learn spatial relationships from scratch through self-attention. This weaker prior requires massive datasets (such as ImageNet-21k or JFT-300M) to learn effective visual representations, though techniques like DeiT distillation and DINOv2 self-supervision can mitigate this requirement.

How does the parameter count scale differently in ViT compared to CNNs?

CNN parameters scale primarily with the number of channels and kernel sizes, remaining relatively modest due to weight sharing across spatial dimensions. ViT parameters scale quadratically with the model dimension (d) because each self-attention layer computes four projection matrices (Q, K, V, O) each of size d × d. This explains why ViT-Base has approximately 86 million parameters compared to ResNet-50's 25 million, despite similar ImageNet performance.

Where can I find the implementation details in the ai-engineering-from-scratch repository?

The ViT implementation, including the patchify function and linear projection logic, is located in phases/07-transformers-deep-dive/09-vision-transformers/code/main.py (lines 17-24 and 55-66). The CNN architectural details, including convolutional hierarchies and residual connections, are documented in phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md. The glossary at glossary/terms.md also defines key concepts like "inductive bias" and "token" used throughout these comparisons.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}