What Makes Vision Transformers (ViT) Different from Traditional CNNs in ai-engineering-from-scratch

Vision Transformers replace convolutional operations with patch-based tokenization and global self-attention, eliminating the strong spatial inductive bias of CNNs while treating images as sequences rather than grids.

The rohitg00/ai-engineering-from-scratch repository demonstrates what makes vision transformers (ViT) different from traditional CNNs by implementing the complete transformer pipeline from scratch using pure NumPy. While CNNs rely on sliding convolutional kernels and hierarchical feature extraction, ViTs split images into fixed-size patches, embed them as tokens, and process them through multi-head self-attention layers—adapting the architecture originally designed for natural language processing directly to computer vision tasks.

Input Processing: Patches vs Convolutions

Traditional CNNs process raw pixels through convolutional kernels that slide across the image grid, exploiting locality and weight sharing. In contrast, the ViT implementation in phases/04-computer-vision/14-vision-transformers/code/main.py splits the image into a regular grid of fixed-size patches (typically 16×16 pixels).

The img_to_patches function reshapes a 224×224 RGB image into 196 patches of 768 dimensions each (16×16×3), transforming the spatial grid into a sequence. Each patch is then flattened and linearly projected to a fixed-dimensional embedding via the linear_proj function, functioning exactly like a word token in NLP rather than a convolved feature map.

Inductive Bias and Spatial Relationships

CNNs encode strong architectural inductive biases including translation invariance, locality, and hierarchical feature extraction. Vision transformers introduce minimal inductive bias, forcing the model to learn spatial relationships solely from training data rather than baked-in assumptions.

To preserve spatial information, the repository adds positional embeddings through the add_positional_emb function. These sinusoidal or learnable vectors are added to patch tokens to encode location, compensating for the transformer's inherent permutation-invariance. Unlike CNNs, which inherently know that adjacent pixels matter due to kernel locality, ViTs must discover spatial structure through attention patterns.

Receptive Field and Computational Complexity

The receptive field represents the most significant architectural divergence. CNNs gradually grow their receptive field with depth, with early layers seeing only small 3×3 or 5×5 neighborhoods. Vision transformers achieve a global receptive field immediately—every token can attend to every other token from the first layer through the multihead_self_attention function.

This global scope comes at a computational cost. While CNNs scale linearly with pixel count O(H·W·K²) for kernel size K, ViTs scale quadratically with the number of patches O(N²), where N equals (H·W)/(patch-size²). Reducing patch size from 16×16 to 8×8 quadruples the number of tokens and increases attention computation by 16×, making the patch size a critical hyperparameter in the transformer_encoder implementation.

The Six-Step ViT Pipeline

The curriculum in phases/04-computer-vision/14-vision-transformers/code/main.py walks through a complete transformer pipeline that unifies vision and language processing:

1. Patchification – The img_to_patches function divides the image into fixed-size patches and flattens them.
2. Linear embedding – A learnable projection maps each flattened patch to a token vector (D=768).
3. [CLS] token – A special classification token is prepended to the sequence, as in BERT.
4. Positional embeddings – Sinusoidal encodings are added via add_positional_emb to retain spatial order.
5. Transformer encoder – Six layers of multihead_self_attention processes the token sequence with residual connections.
6. Classification head – The final hidden state of the [CLS] token feeds into a linear layer for prediction.

This implementation demonstrates how the same architecture processes both text tokens and image patches, illustrating the patch-token primitive that unifies multimodal learning.

NumPy Implementation from Scratch

Below is the self-contained implementation from the repository, demonstrating the full forward pass without external deep learning frameworks:

import numpy as np

def img_to_patches(img, patch_sz):
    """Split H×W×C image into (H/ps)*(W/ps) patches."""
    H, W, C = img.shape
    ph, pw = patch_sz, patch_sz
    patches = img.reshape(H // ph, ph, W // pw, pw, C)
    patches = patches.transpose(0, 2, 1, 3, 4)
    return patches.reshape(-1, ph * pw * C)          # (N, D)

def linear_proj(patches, dim):
    """Learnable linear projection from patch dim → token dim."""
    W = np.random.randn(patches.shape[1], dim) * 0.02
    b = np.zeros(dim)
    return patches @ W + b                         # (N, dim)

def add_positional_emb(tokens):
    N, D = tokens.shape
    pos = np.arange(N)[:, None] / np.power(10000, np.arange(D)[None, :] / D)
    pos = np.sin(pos)                               # sinusoidal encoding

    return tokens + pos

def multihead_self_attention(x, heads=4):
    """Very rough multi‑head self‑attention (no masking)."""
    N, D = x.shape
    head_dim = D // heads
    Wq = np.random.randn(D, D) * 0.02
    Wk = np.random.randn(D, D) * 0.02
    Wv = np.random.randn(D, D) * 0.02
    Q, K, V = x @ Wq, x @ Wk, x @ Wv
    Q, K, V = Q.reshape(N, heads, head_dim), K.reshape(N, heads, head_dim), V.reshape(N, heads, head_dim)
    scores = (Q @ K.transpose(0, 2, 1)) / np.sqrt(head_dim)   # (N, heads, N)

    attn = np.exp(scores - scores.max(axis=-1, keepdims=True))
    attn /= attn.sum(axis=-1, keepdims=True)
    out = (attn @ V).reshape(N, D)
    return out

def transformer_encoder(tokens, layers=6):
    x = tokens
    for _ in range(layers):
        # Self‑attention + residual

        attn = multihead_self_attention(x)
        x = x + attn
        # Feed‑forward + residual (simple linear + ReLU)

        ff = np.maximum(0, x @ np.random.randn(x.shape[1], x.shape[1]) * 0.02)
        x = x + ff
    return x

# -------------------------------------------------

# Example usage on a dummy 224×224 RGB image

img = np.random.randn(224, 224, 3)                # (H, W, C)

patches = img_to_patches(img, patch_sz=16)        # (196, 768)

tokens = linear_proj(patches, dim=768)            # (196, 768)

cls_token = np.zeros((1, 768))                   # learnable in practice

tokens = np.concatenate([cls_token, tokens], 0)  # (197, 768)

tokens = add_positional_emb(tokens)
encoded = transformer_encoder(tokens)              # (197, 768)

logits = encoded[0] @ np.random.randn(768, 10)    # classification head (10 classes)

print("Logits shape:", logits.shape)

Key Curriculum Files

The repository provides comprehensive coverage through these specific files:

• phases/04-computer-vision/14-vision-transformers/docs/en.md – Theoretical background explaining why ViT replaces convolutions
• phases/04-computer-vision/14-vision-transformers/code/main.py – Core NumPy implementation with img_to_patches and transformer_encoder
• phases/04-computer-vision/14-vision-transformers/code/tests/test_main.py – Unit tests validating the encoder functionality
• phases/04-computer-vision/14-vision-transformers/quiz.json – Conceptual checks on architectural differences

Summary

• ViTs tokenize images into patches (16×16) projected to embeddings, while CNNs apply sliding kernels over raw pixels
• Global attention provides immediate full receptive fields versus CNNs' gradual expansion, but costs O(N²) versus O(H·W·K²)
• Minimal inductive bias requires ViTs to learn spatial relationships from data, typically needing larger datasets (ImageNet-1k+) or techniques like register tokens
• Architecture unification allows the same transformer blocks to process both vision patches and language tokens
• Implementation clarity in ai-engineering-from-scratch demonstrates these concepts using only NumPy operations

Frequently Asked Questions

Why do Vision Transformers need more data than CNNs?

CNNs incorporate strong architectural priors like translation invariance and locality through their convolutional structure, allowing them to perform well on modest datasets. Vision transformers lack these built-in assumptions, requiring significantly more training data (typically ImageNet-1k or larger) to learn spatial relationships from scratch. The repository notes that additional techniques like register tokens (Darcet et al., 2023) can help close this data-efficiency gap.

How does the global receptive field in ViTs affect performance?

Unlike CNNs where early layers see only 3×3 pixel neighborhoods, the multihead_self_attention function in ViTs allows every patch to attend to every other patch from the first layer. This enables modeling long-range dependencies immediately—such as relationships between distant object parts—but increases computational cost quadratically with sequence length and can make optimization harder without sufficient data.

What is the role of the [CLS] token in Vision Transformers?

Borrowed from BERT's architecture in NLP, the [CLS] token is a learnable embedding prepended to the patch sequence before entering the transformer_encoder. After processing through all attention layers, the final hidden state corresponding to this token serves as the aggregate image representation, fed directly into the classification head. This design avoids needing global average pooling or other task-specific aggregation mechanisms.

Can Vision Transformers handle dense prediction tasks like segmentation?

While CNNs naturally suit dense prediction through fully-convolutional heads, standard ViTs excel at image-level classification. For dense tasks like segmentation or object detection, Vision Transformers require additional architectural modifications such as patch-wise decoders or specialized heads that upsample the token representations. The ai-engineering-from-scratch curriculum focuses on classification, but the patch-token primitive provides the foundation for these extensions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}