# Implementing CNN Architectures Like ResNet and VGG from Scratch: A Complete PyTorch Guide > Build VGG and ResNet CNN architectures from scratch in PyTorch. Learn to implement convolutional blocks and residual skip connections following the AI For Beginners curriculum. Get the complete guide. - Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners) - Tags: tutorial - Published: 2026-08-02 --- **You can implement VGG-16 and ResNet-18 from scratch using PyTorch by stacking 3×3 convolutional blocks for VGG and using residual skip connections with batch normalization for ResNet, following the canonical patterns defined in the microsoft/AI-For-Beginners curriculum.** Convolutional Neural Networks (CNNs) remain the foundation of modern computer vision, with **VGG** and **ResNet** serving as the most influential architectural blueprints for deep learning practitioners. Building these models from scratch reinforces the mechanics of hierarchical feature extraction and gradient flow that pre-trained APIs often abstract away. This guide walks through complete PyTorch implementations based on the source code in the `microsoft/AI-For-Beginners` repository, specifically referencing the architecture definitions found in [`lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) and the reference implementations in [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py). ## Understanding VGG: Deep Stacking with 3×3 Convolutions ### VGG Architecture Philosophy The **VGG** architecture (Simonyan & Zisserman, 2014) demonstrates that network depth significantly improves representation power when combined with small receptive fields. Rather than using large 7×7 or 11×11 kernels, VGG stacks multiple **3×3 convolution layers** followed by max-pooling, creating a pattern of Conv-3-3 → ReLU → Conv-3-3 → ReLU → MaxPool. According to the curriculum's [`CNN_Architectures.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/CNN_Architectures.md), this design allows the network to learn more complex features while keeping the parameter count manageable compared to larger single-layer kernels. VGG-16 contains 13 convolutional layers and 3 fully-connected layers, totaling approximately 138 million parameters. ### VGG-16 Implementation from Scratch The following implementation mirrors the structure described in the **07-ConvNets** lesson, using a modular `VGGBlock` class that encapsulates the repeated convolution-maxpool pattern: ```python import torch import torch.nn as nn class VGGBlock(nn.Module): """A VGG block: N convolution layers followed by max-pool.""" def __init__(self, in_channels, out_channels, n_convs): super().__init__() layers = [] for _ in range(n_convs): layers.append(nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1)) layers.append(nn.ReLU(inplace=True)) in_channels = out_channels layers.append(nn.MaxPool2d(kernel_size=2, stride=2)) self.block = nn.Sequential(*layers) def forward(self, x): return self.block(x) class VGG16(nn.Module): """VGG-16 architecture, exactly as described in the curriculum.""" def __init__(self, num_classes=1000): super().__init__() self.features = nn.Sequential( VGGBlock(3, 64, 2), # conv1_1, conv1_2 VGGBlock(64, 128, 2), # conv2_1, conv2_2 VGGBlock(128, 256, 3), # conv3_1-3 VGGBlock(256, 512, 3), # conv4_1-3 VGGBlock(512, 512, 3) # conv5_1-3 ) self.classifier = nn.Sequential( nn.Flatten(), nn.Linear(512*7*7, 4096), nn.ReLU(True), nn.Dropout(), nn.Linear(4096, 4096), nn.ReLU(True), nn.Dropout(), nn.Linear(4096, num_classes) ) def forward(self, x): x = self.features(x) return self.classifier(x) # quick sanity check model = VGG16(num_classes=10) print(model) # prints the full architecture ``` The `VGGBlock` abstraction allows you to adjust the depth of each stage by changing the `n_convs` parameter, making it trivial to modify the architecture for VGG-19 variants (which use 4 convolutions in the final stages instead of 3). ## Understanding ResNet: Learning Residual Functions ### The Residual Block Explained **ResNet** (He et al., 2015) solves the vanishing gradient problem in very deep networks by introducing **residual blocks** that learn a residual mapping δ(x) rather than the direct mapping H(x). The core innovation is the **identity shortcut connection** that skips one or more layers, formulated as `output = F(x) + x`. As implemented in the curriculum's [`CNN_Architectures.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/CNN_Architectures.md), a basic residual block follows the pattern: `x → Conv → BN → ReLU → Conv → BN → add(x) → ReLU`. The shortcut allows gradients to flow directly through the network during backpropagation, enabling successful training of networks with 50, 101, or even 152 layers. ### ResNet-18 Implementation from Scratch This implementation uses the `BasicBlock` (two 3×3 conv layers) suitable for ResNet-18 and ResNet-34, as distinct from the bottleneck blocks used in deeper variants: ```python import torch import torch.nn as nn import torch.nn.functional as F class BasicBlock(nn.Module): """ResNet basic block with two 3×3 conv layers.""" expansion = 1 # no channel expansion for basic block def __init__(self, in_planes, out_planes, stride=1): super().__init__() self.conv1 = nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False) self.bn1 = nn.BatchNorm2d(out_planes) self.conv2 = nn.Conv2d(out_planes, out_planes, kernel_size=3, stride=1, padding=1, bias=False) self.bn2 = nn.BatchNorm2d(out_planes) # shortcut (identity) connection self.shortcut = nn.Sequential() if stride != 1 or in_planes != out_planes: # match dimensions via 1×1 conv self.shortcut = nn.Sequential( nn.Conv2d(in_planes, out_planes, kernel_size=1, stride=stride, bias=False), nn.BatchNorm2d(out_planes) ) def forward(self, x): out = F.relu(self.bn1(self.conv1(x))) out = self.bn2(self.conv2(out)) out += self.shortcut(x) # <-- residual addition return F.relu(out) class ResNet(nn.Module): """ResNet-18 built from the basic block defined above.""" def __init__(self, block, num_blocks, num_classes=1000): super().__init__() self.in_planes = 64 self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False) self.bn1 = nn.BatchNorm2d(64) self.pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) # each layer stacks `num_blocks[i]` blocks self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1) self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2) self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2) self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2) self.fc = nn.Linear(512*block.expansion, num_classes) def _make_layer(self, block, out_planes, blocks, stride): strides = [stride] + [1]*(blocks-1) layers = [] for s in strides: layers.append(block(self.in_planes, out_planes, s)) self.in_planes = out_planes * block.expansion return nn.Sequential(*layers) def forward(self, x): out = F.relu(self.bn1(self.conv1(x))) out = self.pool(out) out = self.layer1(out) out = self.layer2(out) out = self.layer3(out) out = self.layer4(out) out = F.avg_pool2d(out, 4) # global average pool out = out.view(out.size(0), -1) return self.fc(out) def ResNet18(num_classes=10): return ResNet(BasicBlock, [2,2,2,2], num_classes) # instantiate and count parameters model = ResNet18() print(model) print(f'Total parameters: {sum(p.numel() for p in model.parameters())/1e6:.2f} M') ``` The critical implementation detail is the dimension matching in the shortcut connection: when `stride != 1` or channel dimensions change, a **1×1 convolution** projects the input to the correct shape before the residual addition. ## Practical Transfer Learning with Pre-trained Models While implementing from scratch builds intuition, production workflows typically leverage pre-trained weights from ImageNet. The `microsoft/AI-For-Beginners` curriculum demonstrates this in `lessons/4-ComputerVision/08-TransferLearning/TransferLearningTF.ipynb` and `TransferLearningPyTorch.ipynb`. ### Loading VGG and ResNet in Keras The following pattern from the TensorFlow transfer learning notebook shows how to freeze the convolutional base of VGG-16 and attach a custom classification head: ```python from tensorflow import keras # Load pre-trained VGG-16 (include_top=False) and freeze conv layers vgg = keras.applications.VGG16(weights='imagenet', include_top=False, input_shape=(224,224,3)) vgg.trainable = False # freeze # Attach a new classifier head model = keras.Sequential([ vgg, keras.layers.GlobalAveragePooling2D(), keras.layers.Dense(256, activation='relu'), keras.layers.Dropout(0.5), keras.layers.Dense(10, activation='softmax') ]) model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy']) model.summary() ``` For ResNet, simply substitute `keras.applications.ResNet50` following the same pattern. The `include_top=False` parameter removes the final fully-connected layers, allowing you to customize the output dimensions for your specific classification task. ### PyTorch Transfer Learning Workflow In PyTorch, the equivalent workflow uses `torchvision.models`, loading pre-trained weights and modifying the final fully-connected layer: ```python import torchvision.models as models # Load pre-trained ResNet-18 model = models.resnet18(weights='IMAGENET1K_V1') # Freeze all layers for param in model.parameters(): param.requires_grad = False # Replace the final FC layer for custom number of classes num_features = model.fc.in_features model.fc = nn.Linear(num_features, 10) # 10 custom classes ``` This approach preserves the learned feature extractors while adapting the classifier to your domain, significantly reducing training time and data requirements compared to training from scratch. ## Summary - **VGG architectures** rely on deep stacking of small 3×3 convolutions with max-pooling, implemented through modular blocks that repeat Conv-ReLU patterns before downsampling. - **ResNet architectures** use residual blocks with skip connections that add the input to the convolutional output, solving vanishing gradients and enabling networks with 50+ layers. - Both architectures can be built from scratch in PyTorch using `nn.Sequential` containers and custom `BasicBlock` or `VGGBlock` classes, as demonstrated in [`lessons/4-ComputerVision/07-ConvNets/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/pytorchcv.py). - For production use, leverage pre-trained weights via `keras.applications` or `torchvision.models`, freezing the convolutional base and replacing the classification head to adapt to new tasks. - The complete implementations and theoretical explanations are available in the **07-ConvNets** and **08-TransferLearning** folders of the `microsoft/AI-For-Beginners` repository. ## Frequently Asked Questions ### What is the main difference between VGG and ResNet architectures? **VGG** uses a straightforward sequential design where each layer feeds directly into the next, relying on depth (16-19 layers) and small 3×3 filters to capture complex patterns. **ResNet** introduces residual skip connections that allow the network to learn residual functions, effectively enabling the training of much deeper networks (50-152 layers) without suffering from vanishing gradients. While VGG-16 contains approximately 138 million parameters, ResNet-50 achieves better accuracy with only 25 million parameters through the use of bottleneck blocks and efficient residual learning. ### Why does ResNet use 1×1 convolutions in the shortcut connection? The **1×1 convolution** serves as a projection shortcut that matches dimensions when the input and output of a residual block have different spatial sizes or channel depths. When `stride != 1` or when the number of input channels differs from output channels (controlled by the `expansion` factor), the 1×1 conv projects the input `x` to the same shape as `F(x)` so they can be added element-wise in the residual connection. Without this projection, the tensor addition in `out += self.shortcut(x)` would raise a dimension mismatch error. ### How do I choose between BasicBlock and BottleneckBlock for ResNet? **BasicBlock** (two 3×3 conv layers) is used for shallower ResNet variants like ResNet-18 and ResNet-34, as shown in the implementation above. **BottleneckBlock** (1×1 → 3×3 → 1×1 conv layers) is used for deeper networks like ResNet-50, ResNet-101, and ResNet-152 because it reduces computational cost by first projecting channels to a lower dimension (256 → 64), performing the 3×3 convolution, then expanding back. For custom implementations with fewer than 50 layers, BasicBlock provides sufficient capacity without unnecessary complexity. ### Where can I find the official notebooks for these implementations? The canonical references are located in the `microsoft/AI-For-Beginners` repository under `lessons/4-ComputerVision/`. The **07-ConvNets** folder contains [`CNN_Architectures.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/CNN_Architectures.md) (architectural theory) and [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py) (reference PyTorch code), while the **08-TransferLearning** folder contains `TransferLearningTF.ipynb` (Keras implementation) and `TransferLearningPyTorch.ipynb` (PyTorch implementation). These notebooks demonstrate both building from scratch and using pre-trained weights for transfer learning.