How to Build CNN Architectures for Image Classification: From Single Layers to LeNet

Build CNN architectures for image classification by stacking convolutional layers with pooling operations to extract spatial hierarchies, followed by fully-connected layers for classification, using frameworks like TensorFlow/Keras or PyTorch as demonstrated in the Microsoft AI-For-Beginners repository.

Convolutional Neural Networks (CNNs) are the foundation of modern computer vision systems. In the microsoft/AI-For-Beginners repository, comprehensive notebooks located at lessons/4-ComputerVision/07-ConvNets/ demonstrate how to construct these architectures from scratch. This guide walks through the core components, proven architectural patterns, and implementation details extracted directly from the source code.

Core Components of CNN Architectures

Convolutional Filters and Feature Maps

The fundamental building block is the convolutional filter, a small $k \times k$ matrix (typically $5 \times 5$ or $3 \times 3$) that slides across the input image to compute weighted sums of local pixel neighborhoods. As implemented in ConvNetsTF.ipynb, the keras.layers.Conv2D layer (or nn.Conv2d in PyTorch as shown in ConvNetsPyTorch.ipynb) applies these trainable filters to produce feature maps that capture edges, textures, and shapes. During training, the network learns which filters best discriminate between classes.

Activation and Pooling Operations

After each convolution, a ReLU activation function introduces non-linearity, allowing the network to learn complex patterns. To reduce spatial dimensions and computational load while preserving salient information, pooling layers are inserted between convolutions. The repository demonstrates both MaxPooling (keeping the strongest activation in each window) and AveragePooling (computing the mean), with MaxPooling being the preferred choice for feature preservation.

Classification Head

Before the final prediction, the 3-D feature map $(C, H, W)$ must be converted to a 1-D vector via a Flatten layer (TensorFlow) or view() operation (PyTorch). This flattened representation feeds into fully-connected (Dense) layers that map the learned features to class scores. For classification tasks, the final layer outputs logits corresponding to the number of target classes.

Three Proven CNN Architectures

The repository provides three scalable designs, each illustrating different trade-offs between complexity and performance.

Single-Layer CNN for Rapid Prototyping

The simplest architecture uses one convolutional layer followed immediately by flattening and a dense output layer. With approximately 52,000 parameters (9 filters of $5 \times 5$ plus bias terms), this design is ideal for small datasets like MNIST or quick prototyping. It demonstrates the basic mechanics of feature extraction without the complexity of deep hierarchies.

Two-Layer CNN with Pooling

Adding a second convolutional layer with intervening MaxPooling dramatically reduces the parameter count to roughly 8,500 while enabling deeper feature hierarchies. This "pyramid" design—implemented in both TensorFlow and PyTorch notebooks—alternates between convolution and pooling to capture increasingly abstract patterns while limiting overfitting through spatial dimension reduction.

LeNet-Style Network for CIFAR-10

For 3-channel color images, the repository implements a classic LeNet-style architecture containing three convolutional layers, two pooling stages, and two dense layers. This configuration uses approximately 59,000 parameters and accepts inputs of shape $32 \times 32 \times 3$, making it suitable for datasets like CIFAR-10. The additional convolutional layer enables the network to learn hierarchical combinations of low-level features.

Framework-Specific Implementation Examples

TensorFlow/Keras Single-Layer CNN

As defined in ConvNetsTF.ipynb, this minimal implementation trains a single-layer CNN on MNIST:

import tensorflow as tf
from tensorflow import keras

# Load MNIST (grayscale) and add channel dimension

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train[..., tf.newaxis] / 255.0
x_test  = x_test[..., tf.newaxis] / 255.0

model = keras.models.Sequential([
    keras.layers.Conv2D(filters=9, kernel_size=(5,5), activation='relu',
                       input_shape=(28,28,1)),
    keras.layers.Flatten(),
    keras.layers.Dense(10)                     # logits

])

model.compile(loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5, validation_data=(x_test, y_test))

PyTorch Two-Layer CNN with Pooling

The ConvNetsPyTorch.ipynb notebook demonstrates a more efficient architecture using two convolutional layers with pooling:

import torch, torch.nn as nn, torch.nn.functional as F
from torchinfo import summary

class TwoLayerCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, 5)   # (1,28,28) -> (10,24,24)

        self.pool  = nn.MaxPool2d(2, 2)    # -> (10,12,12)

        self.conv2 = nn.Conv2d(10, 20, 5)  # -> (20,8,8)

        self.fc    = nn.Linear(20*4*4, 10) # flatten to 10 classes

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))   # (20,4,4)

        x = x.view(-1, 20*4*4)
        return F.log_softmax(self.fc(x), dim=1)

net = TwoLayerCNN()
summary(net, input_size=(1,1,28,28))

LeNet Architecture in PyTorch

For CIFAR-10 classification, the repository implements the following LeNet-style network:

class LeNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)        # (3,32,32) -> (6,28,28)

        self.pool  = nn.MaxPool2d(2)          # -> (6,14,14)

        self.conv2 = nn.Conv2d(6, 16, 5)       # -> (16,10,10)

        self.conv3 = nn.Conv2d(16, 120, 5)     # -> (120,1,1)

        self.fc1   = nn.Linear(120, 64)
        self.fc2   = nn.Linear(64, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = F.relu(self.conv3(x))
        x = x.view(-1, 120)
        x = F.relu(self.fc1(x))
        return self.fc2(x)

net = LeNet()
summary(net, input_size=(1,3,32,32))

Why CNNs Outperform Fully-Connected Networks

Unlike dense networks where each weight connects to every pixel, convolutional layers employ weight sharing across spatial dimensions. This architectural constraint enforces translational invariance—the network recognizes patterns regardless of their position in the image—while drastically reducing the total number of trainable parameters. As illustrated in the comparison between single-layer (~52K parameters) and two-layer (~8.5K parameters) architectures, this efficiency enables robust generalization even with limited training data.

Step-by-Step Guide to Designing Your CNN

Follow this systematic approach to construct effective CNN architectures for image classification:

1. Choose input shape: Define dimensions based on your dataset—28×28×1 for grayscale MNIST or 32×32×3 for CIFAR-10 color images.
2. Select filter configuration: Begin with 8-16 filters of size $3 \times 3$ or $5 \times 5$, adjusting based on the complexity of visual features in your domain.
3. Insert pooling layers: Apply MaxPool2d or MaxPooling2D after every 1-2 convolutions to shrink spatial dimensions and prevent overfitting.
4. Add the dense head: Flatten the final feature map and connect to fully-connected layers with units matching your number of classes.
5. Compile and train: Use SparseCategoricalCrossentropy (TensorFlow) or CrossEntropyLoss (PyTorch) with the Adam optimizer, monitoring validation accuracy to detect overfitting.

Summary

• Convolutional layers use small, trainable filters ($5 \times 5$ or $3 \times 3$) to extract spatial features while sharing weights across the image, drastically reducing parameter counts compared to dense networks.
• Pooling operations (particularly MaxPooling) reduce spatial dimensions between convolutions, enabling deeper architectures like the two-layer CNN (~8.5K parameters) that outperform single-layer designs (~52K parameters) through hierarchical feature learning.
• LeNet-style architectures with three convolutions and two pooling stages provide robust baselines for color image classification on datasets like CIFAR-10 using approximately 59K parameters.
• Framework implementations in ConvNetsTF.ipynb and ConvNetsPyTorch.ipynb demonstrate that the architectural principles remain consistent across TensorFlow/Keras and PyTorch, differing only in API syntax.

Frequently Asked Questions

What is the difference between MaxPooling and AveragePooling in CNNs?

MaxPooling retains the maximum activation value within each pooling window, preserving the strongest detected features and providing translation invariance. AveragePooling computes the mean of all values in the window, which smooths the feature map but may dilute strong signals. According to the source notebooks, MaxPooling is generally preferred for classification tasks because it maintains the most salient spatial information.

How many parameters does a typical CNN for MNIST classification require?

A single-layer CNN for MNIST requires approximately 52,000 parameters (primarily from 9 filters of size $5 \times 5$ plus bias terms). However, adding a second convolutional layer with pooling reduces this to roughly 8,500 parameters while improving performance through deeper feature hierarchies. This efficiency gain demonstrates why multi-layer architectures are preferred despite their increased depth.

Why use 5x5 filters instead of 3x3 in beginner CNN architectures?

While modern architectures often use $3 \times 3$ filters for computational efficiency, the Microsoft AI-For-Beginners repository employs $5 \times 5$ filters in introductory examples because they capture larger spatial contexts with fewer layers. This makes the feature extraction process more interpretable for beginners, though production systems typically stack multiple $3 \times 3$ layers to achieve larger receptive fields with fewer total parameters.

Can I use these CNN architectures for datasets other than MNIST and CIFAR-10?

Yes, these architectures serve as templates for various image classification tasks. For larger or higher-resolution images (e.g., ImageNet), you should increase the depth (more convolutional layers), adjust filter counts, and potentially add regularization techniques like dropout or batch normalization. The fundamental pattern—convolution, activation, pooling, followed by dense layers—remains applicable across domains, though input shapes and network capacity must scale with dataset complexity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}