# How to Implement Convolutional Neural Networks from Scratch vs Using Frameworks

> Learn to implement convolutional neural networks from scratch with NumPy or leverage PyTorch and TensorFlow for efficient production development. Understand the trade-offs for your AI projects.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: deep-dive
- Published: 2026-08-03

---

**Implement CNNs from scratch using NumPy for educational clarity, or use PyTorch/TensorFlow frameworks like `nn.Conv2d` and `tf.nn.conv2d` for production performance and automatic differentiation.**

A Convolutional Neural Network (CNN) is the backbone of modern computer vision. The **microsoft/AI-For-Beginners** repository provides clear examples of both approaches: manual NumPy implementations that reveal the underlying mathematics, and high-level framework code that powers real-world applications. This guide compares both methods using actual source files from the repository's computer vision lessons.

## Core CNN Operations Explained

Every CNN performs three fundamental operations. Understanding how each is implemented—from scratch versus using frameworks—clarifies when to choose which approach.

### Convolution: The Sliding Window Operation

**Convolution** slides a learnable kernel across input data, computing dot products at each spatial location.

**From Scratch (NumPy):**

You manually implement nested loops or array slicing to extract windows, multiply element-wise with the kernel, and sum the results. You must handle **padding** to preserve dimensions and **stride** to control downsampling yourself.

```python
import numpy as np

def manual_conv2d(image, kernel, stride=1, padding=0):
    # Add padding

    if padding > 0:
        image = np.pad(image, padding, mode='constant')
    
    kh, kw = kernel.shape
    ih, iw = image.shape
    oh = (ih - kh) // stride + 1
    ow = (iw - kw) // stride + 1
    
    output = np.zeros((oh, ow))
    
    # Manual sliding window

    for i in range(0, oh * stride, stride):
        for j in range(0, ow * stride, stride):
            window = image[i:i+kh, j:j+kw]
            output[i//stride, j//stride] = np.sum(window * kernel)
    
    return output

```

**Using PyTorch Framework:**

In [`lessons/4-ComputerVision/07-ConvNets/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/pytorchcv.py), the `plot_convolution` function demonstrates the framework approach. The `nn.Conv2d` layer automatically creates weight tensors, handles stride and padding, and prepares for GPU execution.

```python
import torch
import torch.nn as nn

# Framework handles kernel initialization, stride, padding internally

conv = nn.Conv2d(
    in_channels=1,      # Input feature maps

    out_channels=1,     # Output feature maps  

    kernel_size=3,      # 3x3 kernel

    stride=1,           # Step size

    padding=0,          # Zero-padding

    bias=False          # No bias term

)

# Forward pass computes convolution automatically

output = conv(torch.randn(1, 1, 28, 28))  # (batch, channel, height, width)

```

The repository's `plot_convolution` at lines 103-112 specifically shows copying custom weights into `conv.weight.data` for visualization purposes—demonstrating that framework layers expose their parameters when needed.

**Using TensorFlow Framework:**

In [`lessons/4-ComputerVision/07-ConvNets/tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/tfcv.py), the TensorFlow implementation at lines 16-19 uses `tf.nn.conv2d` for explicit control over the convolution operation:

```python
import tensorflow as tf

kernel = tf.random.normal([3, 3, 1, 1])  # [height, width, in_channels, out_channels]

image = tf.random.normal([1, 28, 28, 1]) # [batch, height, width, channels]

# Explicit convolution with framework optimization

conv = tf.nn.conv2d(
    image,
    kernel,
    strides=[1, 1, 1, 1],  # [batch, height, width, channels]

    padding='SAME'         # Automatic padding to preserve size

)

```

### Activation Functions: Adding Non-Linearity

**From Scratch:**

```python

# ReLU activation: max(0, x)

activated = np.maximum(0, conv_output)

```

**PyTorch Framework:**

```python
relu = nn.ReLU()
activated = relu(conv_output)

```

**TensorFlow Framework:**

```python
activated = tf.nn.relu(conv_output)

```

### Pooling: Spatial Dimension Reduction

| Approach | Max Pooling Implementation | Average Pooling Implementation |
|----------|---------------------------|-------------------------------|
| **Scratch** | Extract window maxima with nested loops or `np.max` over strided windows | Compute `np.mean` over each pooling window |
| **PyTorch** | `nn.MaxPool2d(kernel_size=2, stride=2)` | `nn.AvgPool2d(kernel_size=2, stride=2)` |
| **TensorFlow** | `tf.nn.max_pool` with `ksize` and `strides` parameters | `tf.nn.avg_pool` with identical parameter structure |

## Complete Implementation Comparison

### Building a CNN from Scratch

A full scratch implementation requires manual orchestration of every training component:

```python
import numpy as np

class ScratchCNN:
    def __init__(self):
        # Manual weight initialization (He/Kaiming or Xavier)

        self.conv1_w = np.random.randn(6, 1, 5, 5) * np.sqrt(2.0 / 25)
        self.conv1_b = np.zeros(6)
        self.fc_w = np.random.randn(6 * 24 * 24, 10) * 0.01
        self.fc_b = np.zeros(10)
    
    def forward(self, x):
        # Manual convolution

        conv1 = self.manual_conv(x, self.conv1_w) + self.conv1_b
        # Manual ReLU

        relu1 = np.maximum(0, conv1)
        # Manual pooling

        pool1 = self.manual_maxpool(relu1)
        # Flatten and fully connected

        flat = pool1.reshape(pool1.shape[0], -1)
        out = flat @ self.fc_w + self.fc_b
        return out
    
    def backward(self, grad_output, learning_rate):
        # Manual gradient computation via chain rule

        # Manual parameter updates

        pass

```

**Challenges of scratch implementation:**

- Must derive and code gradients for every operation
- No automatic GPU acceleration
- Debugging requires verifying each layer's output shape manually
- Numeric stability issues (vanishing/exploding gradients) must be handled explicitly

### Building a CNN with PyTorch Framework

The repository's [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py) shows the concise framework approach:

```python
import torch
import torch.nn as nn
import torch.nn.functional as F

class FrameworkCNN(nn.Module):
    def __init__(self):
        super().__init__()
        # Layers automatically create and initialize parameters

        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc = nn.Linear(6 * 12 * 12, 10)  # Size computed automatically

    
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))  # Chain operations cleanly

        x = x.view(x.size(0), -1)             # Flatten

        x = self.fc(x)
        return x

# Automatic differentiation handles gradients

model = FrameworkCNN()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()

# Training loop: framework computes gradients automatically

for data, target in dataloader:
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()        # Autograd computes all gradients

    optimizer.step()       # Parameters updated automatically

```

### Building a CNN with TensorFlow Framework

The repository's [`tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/tfcv.py) demonstrates TensorFlow's similar abstraction:

```python
import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(6, 5, activation='relu', input_shape=(28, 28, 1)),
    tf.keras.layers.MaxPooling2D(2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(10)
])

# Framework handles entire training pipeline

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

model.fit(x_train, y_train, epochs=5)  # One line for full training loop

```

## Performance and Practical Trade-offs

| Aspect | Scratch (NumPy) | PyTorch/TensorFlow |
|--------|---------------|-------------------|
| **Lines of code** | ~500+ for basic CNN | ~20-50 with same architecture |
| **GPU acceleration** | Manual CUDA coding required | Single line: `.to('cuda')` or automatic |
| **Gradient computation** | Derive and implement chain rule | Automatic differentiation (autograd) |
| **Debugging insight** | Maximum—see every intermediate value | Moderate—use hooks or print statements |
| **Production deployment** | Impractical | TorchScript, ONNX, TensorFlow Serving |
| **Custom operations** | Full flexibility | Possible but requires C++ extensions |

## When to Choose Each Approach

**Choose scratch implementation when:**

- Learning fundamentals of how CNNs actually work
- Implementing novel operations not available in frameworks
- Teaching or explaining the mathematics to others
- Working in constrained environments without framework installations

**Choose framework implementation when:**

- Building production computer vision systems
- Researching new architectures requiring rapid iteration
- Leveraging pre-trained models (transfer learning) as shown in [`lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py)
- Needing distributed training across multiple GPUs
- Deploying to edge devices via optimization tools (TensorRT, Core ML)

## Repository Source Files Reference

| File | Purpose | Key Components |
|------|---------|--------------|
| [`lessons/4-ComputerVision/07-ConvNets/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/pytorchcv.py) | PyTorch convolution demonstration | `plot_convolution()`, `nn.Conv2d` usage, visualization utilities |
| [`lessons/4-ComputerVision/07-ConvNets/tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/tfcv.py) | TensorFlow convolution demonstration | `tf.nn.conv2d`, kernel customization, training workflow |
| [`lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py) | Advanced framework usage | Pre-trained model loading, feature extraction, fine-tuning |

## Summary

- **Convolution from scratch** requires manual sliding-window dot products, explicit padding and stride handling, and self-derived gradients—valuable for understanding but impractical for scale.

- **Framework implementations** in PyTorch (`nn.Conv2d`) and TensorFlow (`tf.nn.conv2d`) abstract convolution into parameterized layers with automatic differentiation, GPU acceleration, and production-ready deployment options.

- The **microsoft/AI-For-Beginners** repository provides working examples of both: [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py) and [`tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/tfcv.py) demonstrate framework abstractions that would replace hundreds of lines of manual NumPy code.

- For educational purposes, implement at least one convolution layer manually; for all practical applications, use established frameworks.

## Frequently Asked Questions

### Can I mix scratch and framework implementations in the same model?

Yes, PyTorch and TensorFlow both support custom operations. In PyTorch, use `torch.autograd.Function` to define forward and backward passes for custom layers. In TensorFlow, use `tf.custom_gradient` or create layers subclassing `tf.keras.layers.Layer`. This lets you insert manually-coded convolutions into otherwise framework-standard pipelines.

### How much faster is framework GPU acceleration compared to NumPy?

Typical speedups range from 10-100x for CNN training, depending on model size and hardware. A ResNet-50 training run that takes hours on CPU-based NumPy completes in minutes on a GPU with PyTorch or TensorFlow, due to cuDNN-optimized convolution kernels and parallel batch processing.

### Does implementing from scratch help with debugging framework code?

Absolutely. Understanding how `im2col` transforms, padding calculations, and backpropagation through convolutions work manually makes you far more effective at diagnosing shape mismatches, vanishing gradients, and unexpected outputs in framework code. The repository's lessons are structured to build this intuition first.