Implementing CNN Architectures Like ResNet and VGG from Scratch: A Complete PyTorch Guide

You can implement VGG-16 and ResNet-18 from scratch using PyTorch by stacking 3×3 convolutional blocks for VGG and using residual skip connections with batch normalization for ResNet, following the canonical patterns defined in the microsoft/AI-For-Beginners curriculum.

Convolutional Neural Networks (CNNs) remain the foundation of modern computer vision, with VGG and ResNet serving as the most influential architectural blueprints for deep learning practitioners. Building these models from scratch reinforces the mechanics of hierarchical feature extraction and gradient flow that pre-trained APIs often abstract away. This guide walks through complete PyTorch implementations based on the source code in the microsoft/AI-For-Beginners repository, specifically referencing the architecture definitions found in lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md and the reference implementations in pytorchcv.py.

Understanding VGG: Deep Stacking with 3×3 Convolutions

VGG Architecture Philosophy

The VGG architecture (Simonyan & Zisserman, 2014) demonstrates that network depth significantly improves representation power when combined with small receptive fields. Rather than using large 7×7 or 11×11 kernels, VGG stacks multiple 3×3 convolution layers followed by max-pooling, creating a pattern of Conv-3-3 → ReLU → Conv-3-3 → ReLU → MaxPool.

According to the curriculum's CNN_Architectures.md, this design allows the network to learn more complex features while keeping the parameter count manageable compared to larger single-layer kernels. VGG-16 contains 13 convolutional layers and 3 fully-connected layers, totaling approximately 138 million parameters.

VGG-16 Implementation from Scratch

The following implementation mirrors the structure described in the 07-ConvNets lesson, using a modular VGGBlock class that encapsulates the repeated convolution-maxpool pattern:

import torch
import torch.nn as nn

class VGGBlock(nn.Module):
    """A VGG block: N convolution layers followed by max-pool."""
    def __init__(self, in_channels, out_channels, n_convs):
        super().__init__()
        layers = []
        for _ in range(n_convs):
            layers.append(nn.Conv2d(in_channels, out_channels,
                                   kernel_size=3, padding=1))
            layers.append(nn.ReLU(inplace=True))
            in_channels = out_channels
        layers.append(nn.MaxPool2d(kernel_size=2, stride=2))
        self.block = nn.Sequential(*layers)

    def forward(self, x):
        return self.block(x)

class VGG16(nn.Module):
    """VGG-16 architecture, exactly as described in the curriculum."""
    def __init__(self, num_classes=1000):
        super().__init__()
        self.features = nn.Sequential(
            VGGBlock(3, 64, 2),   # conv1_1, conv1_2

            VGGBlock(64, 128, 2),   # conv2_1, conv2_2

            VGGBlock(128, 256, 3),  # conv3_1-3

            VGGBlock(256, 512, 3),  # conv4_1-3

            VGGBlock(512, 512, 3)   # conv5_1-3

        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(512*7*7, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        return self.classifier(x)

# quick sanity check

model = VGG16(num_classes=10)
print(model)                       # prints the full architecture

The VGGBlock abstraction allows you to adjust the depth of each stage by changing the n_convs parameter, making it trivial to modify the architecture for VGG-19 variants (which use 4 convolutions in the final stages instead of 3).

Understanding ResNet: Learning Residual Functions

The Residual Block Explained

ResNet (He et al., 2015) solves the vanishing gradient problem in very deep networks by introducing residual blocks that learn a residual mapping δ(x) rather than the direct mapping H(x). The core innovation is the identity shortcut connection that skips one or more layers, formulated as output = F(x) + x.

As implemented in the curriculum's CNN_Architectures.md, a basic residual block follows the pattern: x → Conv → BN → ReLU → Conv → BN → add(x) → ReLU. The shortcut allows gradients to flow directly through the network during backpropagation, enabling successful training of networks with 50, 101, or even 152 layers.

ResNet-18 Implementation from Scratch

This implementation uses the BasicBlock (two 3×3 conv layers) suitable for ResNet-18 and ResNet-34, as distinct from the bottleneck blocks used in deeper variants:

import torch
import torch.nn as nn
import torch.nn.functional as F

class BasicBlock(nn.Module):
    """ResNet basic block with two 3×3 conv layers."""
    expansion = 1                     # no channel expansion for basic block

    def __init__(self, in_planes, out_planes, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_planes, out_planes,
                               kernel_size=3, stride=stride,
                               padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_planes)
        self.conv2 = nn.Conv2d(out_planes, out_planes,
                               kernel_size=3, stride=1,
                               padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_planes)

        # shortcut (identity) connection

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != out_planes:
            # match dimensions via 1×1 conv

            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, out_planes,
                          kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(out_planes)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)          # <-- residual addition

        return F.relu(out)

class ResNet(nn.Module):
    """ResNet-18 built from the basic block defined above."""
    def __init__(self, block, num_blocks, num_classes=1000):
        super().__init__()
        self.in_planes = 64

        self.conv1 = nn.Conv2d(3, 64, kernel_size=7,
                               stride=2, padding=3, bias=False)
        self.bn1   = nn.BatchNorm2d(64)
        self.pool  = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        # each layer stacks `num_blocks[i]` blocks

        self.layer1 = self._make_layer(block, 64,  num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)

        self.fc = nn.Linear(512*block.expansion, num_classes)

    def _make_layer(self, block, out_planes, blocks, stride):
        strides = [stride] + [1]*(blocks-1)
        layers = []
        for s in strides:
            layers.append(block(self.in_planes, out_planes, s))
            self.in_planes = out_planes * block.expansion
        return nn.Sequential(*layers)

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.pool(out)
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = F.avg_pool2d(out, 4)                # global average pool

        out = out.view(out.size(0), -1)
        return self.fc(out)

def ResNet18(num_classes=10):
    return ResNet(BasicBlock, [2,2,2,2], num_classes)

# instantiate and count parameters

model = ResNet18()
print(model)
print(f'Total parameters: {sum(p.numel() for p in model.parameters())/1e6:.2f} M')

The critical implementation detail is the dimension matching in the shortcut connection: when stride != 1 or channel dimensions change, a 1×1 convolution projects the input to the correct shape before the residual addition.

Practical Transfer Learning with Pre-trained Models

While implementing from scratch builds intuition, production workflows typically leverage pre-trained weights from ImageNet. The microsoft/AI-For-Beginners curriculum demonstrates this in lessons/4-ComputerVision/08-TransferLearning/TransferLearningTF.ipynb and TransferLearningPyTorch.ipynb.

Loading VGG and ResNet in Keras

The following pattern from the TensorFlow transfer learning notebook shows how to freeze the convolutional base of VGG-16 and attach a custom classification head:

from tensorflow import keras

# Load pre-trained VGG-16 (include_top=False) and freeze conv layers

vgg = keras.applications.VGG16(weights='imagenet', include_top=False,
                              input_shape=(224,224,3))
vgg.trainable = False                               # freeze

# Attach a new classifier head

model = keras.Sequential([
    vgg,
    keras.layers.GlobalAveragePooling2D(),
    keras.layers.Dense(256, activation='relu'),
    keras.layers.Dropout(0.5),
    keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.summary()

For ResNet, simply substitute keras.applications.ResNet50 following the same pattern. The include_top=False parameter removes the final fully-connected layers, allowing you to customize the output dimensions for your specific classification task.

PyTorch Transfer Learning Workflow

In PyTorch, the equivalent workflow uses torchvision.models, loading pre-trained weights and modifying the final fully-connected layer:

import torchvision.models as models

# Load pre-trained ResNet-18

model = models.resnet18(weights='IMAGENET1K_V1')

# Freeze all layers

for param in model.parameters():
    param.requires_grad = False

# Replace the final FC layer for custom number of classes

num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 10)  # 10 custom classes

This approach preserves the learned feature extractors while adapting the classifier to your domain, significantly reducing training time and data requirements compared to training from scratch.

Summary

• VGG architectures rely on deep stacking of small 3×3 convolutions with max-pooling, implemented through modular blocks that repeat Conv-ReLU patterns before downsampling.
• ResNet architectures use residual blocks with skip connections that add the input to the convolutional output, solving vanishing gradients and enabling networks with 50+ layers.
• Both architectures can be built from scratch in PyTorch using nn.Sequential containers and custom BasicBlock or VGGBlock classes, as demonstrated in lessons/4-ComputerVision/07-ConvNets/pytorchcv.py.
• For production use, leverage pre-trained weights via keras.applications or torchvision.models, freezing the convolutional base and replacing the classification head to adapt to new tasks.
• The complete implementations and theoretical explanations are available in the 07-ConvNets and 08-TransferLearning folders of the microsoft/AI-For-Beginners repository.

Frequently Asked Questions

What is the main difference between VGG and ResNet architectures?

VGG uses a straightforward sequential design where each layer feeds directly into the next, relying on depth (16-19 layers) and small 3×3 filters to capture complex patterns. ResNet introduces residual skip connections that allow the network to learn residual functions, effectively enabling the training of much deeper networks (50-152 layers) without suffering from vanishing gradients. While VGG-16 contains approximately 138 million parameters, ResNet-50 achieves better accuracy with only 25 million parameters through the use of bottleneck blocks and efficient residual learning.

Why does ResNet use 1×1 convolutions in the shortcut connection?

The 1×1 convolution serves as a projection shortcut that matches dimensions when the input and output of a residual block have different spatial sizes or channel depths. When stride != 1 or when the number of input channels differs from output channels (controlled by the expansion factor), the 1×1 conv projects the input x to the same shape as F(x) so they can be added element-wise in the residual connection. Without this projection, the tensor addition in out += self.shortcut(x) would raise a dimension mismatch error.

How do I choose between BasicBlock and BottleneckBlock for ResNet?

BasicBlock (two 3×3 conv layers) is used for shallower ResNet variants like ResNet-18 and ResNet-34, as shown in the implementation above. BottleneckBlock (1×1 → 3×3 → 1×1 conv layers) is used for deeper networks like ResNet-50, ResNet-101, and ResNet-152 because it reduces computational cost by first projecting channels to a lower dimension (256 → 64), performing the 3×3 convolution, then expanding back. For custom implementations with fewer than 50 layers, BasicBlock provides sufficient capacity without unnecessary complexity.

Where can I find the official notebooks for these implementations?

The canonical references are located in the microsoft/AI-For-Beginners repository under lessons/4-ComputerVision/. The 07-ConvNets folder contains CNN_Architectures.md (architectural theory) and pytorchcv.py (reference PyTorch code), while the 08-TransferLearning folder contains TransferLearningTF.ipynb (Keras implementation) and TransferLearningPyTorch.ipynb (PyTorch implementation). These notebooks demonstrate both building from scratch and using pre-trained weights for transfer learning.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}