How to Implement Convolutional Neural Networks from Scratch vs Using Frameworks

Implement CNNs from scratch using NumPy for educational clarity, or use PyTorch/TensorFlow frameworks like nn.Conv2d and tf.nn.conv2d for production performance and automatic differentiation.

A Convolutional Neural Network (CNN) is the backbone of modern computer vision. The microsoft/AI-For-Beginners repository provides clear examples of both approaches: manual NumPy implementations that reveal the underlying mathematics, and high-level framework code that powers real-world applications. This guide compares both methods using actual source files from the repository's computer vision lessons.

Core CNN Operations Explained

Every CNN performs three fundamental operations. Understanding how each is implemented—from scratch versus using frameworks—clarifies when to choose which approach.

Convolution: The Sliding Window Operation

Convolution slides a learnable kernel across input data, computing dot products at each spatial location.

From Scratch (NumPy):

You manually implement nested loops or array slicing to extract windows, multiply element-wise with the kernel, and sum the results. You must handle padding to preserve dimensions and stride to control downsampling yourself.

import numpy as np

def manual_conv2d(image, kernel, stride=1, padding=0):
    # Add padding

    if padding > 0:
        image = np.pad(image, padding, mode='constant')
    
    kh, kw = kernel.shape
    ih, iw = image.shape
    oh = (ih - kh) // stride + 1
    ow = (iw - kw) // stride + 1
    
    output = np.zeros((oh, ow))
    
    # Manual sliding window

    for i in range(0, oh * stride, stride):
        for j in range(0, ow * stride, stride):
            window = image[i:i+kh, j:j+kw]
            output[i//stride, j//stride] = np.sum(window * kernel)
    
    return output

Using PyTorch Framework:

In lessons/4-ComputerVision/07-ConvNets/pytorchcv.py, the plot_convolution function demonstrates the framework approach. The nn.Conv2d layer automatically creates weight tensors, handles stride and padding, and prepares for GPU execution.

import torch
import torch.nn as nn

# Framework handles kernel initialization, stride, padding internally

conv = nn.Conv2d(
    in_channels=1,      # Input feature maps

    out_channels=1,     # Output feature maps  

    kernel_size=3,      # 3x3 kernel

    stride=1,           # Step size

    padding=0,          # Zero-padding

    bias=False          # No bias term

)

# Forward pass computes convolution automatically

output = conv(torch.randn(1, 1, 28, 28))  # (batch, channel, height, width)

The repository's plot_convolution at lines 103-112 specifically shows copying custom weights into conv.weight.data for visualization purposes—demonstrating that framework layers expose their parameters when needed.

Using TensorFlow Framework:

In lessons/4-ComputerVision/07-ConvNets/tfcv.py, the TensorFlow implementation at lines 16-19 uses tf.nn.conv2d for explicit control over the convolution operation:

import tensorflow as tf

kernel = tf.random.normal([3, 3, 1, 1])  # [height, width, in_channels, out_channels]

image = tf.random.normal([1, 28, 28, 1]) # [batch, height, width, channels]

# Explicit convolution with framework optimization

conv = tf.nn.conv2d(
    image,
    kernel,
    strides=[1, 1, 1, 1],  # [batch, height, width, channels]

    padding='SAME'         # Automatic padding to preserve size

)

Activation Functions: Adding Non-Linearity

From Scratch:

# ReLU activation: max(0, x)

activated = np.maximum(0, conv_output)

PyTorch Framework:

relu = nn.ReLU()
activated = relu(conv_output)

TensorFlow Framework:

activated = tf.nn.relu(conv_output)

Pooling: Spatial Dimension Reduction

Approach	Max Pooling Implementation	Average Pooling Implementation
Scratch	Extract window maxima with nested loops or np.max over strided windows	Compute np.mean over each pooling window
PyTorch	nn.MaxPool2d(kernel_size=2, stride=2)	nn.AvgPool2d(kernel_size=2, stride=2)
TensorFlow	tf.nn.max_pool with ksize and strides parameters	tf.nn.avg_pool with identical parameter structure

Complete Implementation Comparison

Building a CNN from Scratch

A full scratch implementation requires manual orchestration of every training component:

import numpy as np

class ScratchCNN:
    def __init__(self):
        # Manual weight initialization (He/Kaiming or Xavier)

        self.conv1_w = np.random.randn(6, 1, 5, 5) * np.sqrt(2.0 / 25)
        self.conv1_b = np.zeros(6)
        self.fc_w = np.random.randn(6 * 24 * 24, 10) * 0.01
        self.fc_b = np.zeros(10)
    
    def forward(self, x):
        # Manual convolution

        conv1 = self.manual_conv(x, self.conv1_w) + self.conv1_b
        # Manual ReLU

        relu1 = np.maximum(0, conv1)
        # Manual pooling

        pool1 = self.manual_maxpool(relu1)
        # Flatten and fully connected

        flat = pool1.reshape(pool1.shape[0], -1)
        out = flat @ self.fc_w + self.fc_b
        return out
    
    def backward(self, grad_output, learning_rate):
        # Manual gradient computation via chain rule

        # Manual parameter updates

        pass

Challenges of scratch implementation:

• Must derive and code gradients for every operation
• No automatic GPU acceleration
• Debugging requires verifying each layer's output shape manually
• Numeric stability issues (vanishing/exploding gradients) must be handled explicitly

Building a CNN with PyTorch Framework

The repository's pytorchcv.py shows the concise framework approach:

import torch
import torch.nn as nn
import torch.nn.functional as F

class FrameworkCNN(nn.Module):
    def __init__(self):
        super().__init__()
        # Layers automatically create and initialize parameters

        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc = nn.Linear(6 * 12 * 12, 10)  # Size computed automatically

    
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))  # Chain operations cleanly

        x = x.view(x.size(0), -1)             # Flatten

        x = self.fc(x)
        return x

# Automatic differentiation handles gradients

model = FrameworkCNN()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

# Training loop: framework computes gradients automatically

for data, target in dataloader:
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()        # Autograd computes all gradients

    optimizer.step()       # Parameters updated automatically

Building a CNN with TensorFlow Framework

The repository's tfcv.py demonstrates TensorFlow's similar abstraction:

import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(6, 5, activation='relu', input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(10)
])

# Framework handles entire training pipeline

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

model.fit(x_train, y_train, epochs=5)  # One line for full training loop

Performance and Practical Trade-offs

Aspect	Scratch (NumPy)	PyTorch/TensorFlow
Lines of code	~500+ for basic CNN	~20-50 with same architecture
GPU acceleration	Manual CUDA coding required	Single line: .to('cuda') or automatic
Gradient computation	Derive and implement chain rule	Automatic differentiation (autograd)
Debugging insight	Maximum—see every intermediate value	Moderate—use hooks or print statements
Production deployment	Impractical	TorchScript, ONNX, TensorFlow Serving
Custom operations	Full flexibility	Possible but requires C++ extensions

When to Choose Each Approach

Choose scratch implementation when:

• Learning fundamentals of how CNNs actually work
• Implementing novel operations not available in frameworks
• Teaching or explaining the mathematics to others
• Working in constrained environments without framework installations

Choose framework implementation when:

• Building production computer vision systems
• Researching new architectures requiring rapid iteration
• Leveraging pre-trained models (transfer learning) as shown in lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py
• Needing distributed training across multiple GPUs
• Deploying to edge devices via optimization tools (TensorRT, Core ML)

Repository Source Files Reference

File	Purpose	Key Components
lessons/4-ComputerVision/07-ConvNets/pytorchcv.py	PyTorch convolution demonstration	plot_convolution(), nn.Conv2d usage, visualization utilities
lessons/4-ComputerVision/07-ConvNets/tfcv.py	TensorFlow convolution demonstration	tf.nn.conv2d, kernel customization, training workflow
lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py	Advanced framework usage	Pre-trained model loading, feature extraction, fine-tuning

Summary

• Convolution from scratch requires manual sliding-window dot products, explicit padding and stride handling, and self-derived gradients—valuable for understanding but impractical for scale.

• Framework implementations in PyTorch (nn.Conv2d) and TensorFlow (tf.nn.conv2d) abstract convolution into parameterized layers with automatic differentiation, GPU acceleration, and production-ready deployment options.

• The microsoft/AI-For-Beginners repository provides working examples of both: pytorchcv.py and tfcv.py demonstrate framework abstractions that would replace hundreds of lines of manual NumPy code.

• For educational purposes, implement at least one convolution layer manually; for all practical applications, use established frameworks.

Frequently Asked Questions

Can I mix scratch and framework implementations in the same model?

Yes, PyTorch and TensorFlow both support custom operations. In PyTorch, use torch.autograd.Function to define forward and backward passes for custom layers. In TensorFlow, use tf.custom_gradient or create layers subclassing tf.keras.layers.Layer. This lets you insert manually-coded convolutions into otherwise framework-standard pipelines.

How much faster is framework GPU acceleration compared to NumPy?

Typical speedups range from 10-100x for CNN training, depending on model size and hardware. A ResNet-50 training run that takes hours on CPU-based NumPy completes in minutes on a GPU with PyTorch or TensorFlow, due to cuDNN-optimized convolution kernels and parallel batch processing.

Does implementing from scratch help with debugging framework code?

Absolutely. Understanding how im2col transforms, padding calculations, and backpropagation through convolutions work manually makes you far more effective at diagnosing shape mismatches, vanishing gradients, and unexpected outputs in framework code. The repository's lessons are structured to build this intuition first.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}