# Implementing CNN Architectures from Scratch vs Using Keras/TensorFlow: A Deep Dive

> Learn to implement CNN architectures from scratch or use Keras/TensorFlow. Understand the pros and cons of each approach for your machine learning projects.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: deep-dive
- Published: 2026-08-05

---

**Building CNNs from scratch in PyTorch provides complete control over tensor operations and training loops, while Keras/TensorFlow abstracts boilerplate through high-level APIs like `model.fit`, making each approach optimal for different stages of the machine learning workflow.**

Convolutional Neural Networks (CNNs) form the backbone of modern computer vision, and the microsoft/AI-For-Beginners repository demonstrates two fundamentally different approaches to constructing these architectures. Whether you implement CNN architectures from scratch using PyTorch's explicit layer definitions or leverage the abstraction of Keras/TensorFlow, understanding both paradigms is essential for selecting the right tool for research, education, or production deployment.

## Layer Definition and Weight Control

### Manual Implementation in PyTorch

In [`lessons/4-ComputerVision/07-ConvNets/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/pytorchcv.py), the repository demonstrates explicit instantiation of convolutional layers using `nn.Conv2d`, where developers manually initialize weights and handle the forward pass. This approach exposes the underlying mathematics—you directly manipulate weight tensors through operations like `c.weight.copy_(t)`, granting full control over initialization schemes, regularization, and custom constraints.

### High-Level Abstraction in Keras/TensorFlow

Conversely, the TensorFlow implementation in [`tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/tfcv.py) utilizes `tf.nn.conv2d` or `keras.layers.Conv2D`, where the framework manages kernel tensors internally. While weights remain accessible via `layer.weights`, the API encourages allowing Keras to handle the weight lifecycle, significantly reducing code complexity for standard architectures.

## Training Loop Implementation

### Explicit Training Logic

The PyTorch approach requires manual construction of training logic through functions like `train_epoch`, `validate`, and `train` as seen in [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py). Developers must explicitly iterate over batches, compute gradients with an optimizer, calculate loss, and track accuracy metrics—providing transparency into every optimization step.

### Automated Training with model.fit

Keras eliminates this boilerplate through `model.fit`, which automatically handles batch iteration, loss computation, and metric updates. This abstraction, demonstrated in the repository's examples, dramatically accelerates experimentation cycles while seamlessly integrating with the broader TensorFlow ecosystem including **tf.data pipelines** and **TensorBoard** monitoring.

## Practical Code Examples

### Building a CNN from Scratch in PyTorch

The following implementation from the repository shows explicit layer definition and manual training:

```python
import torch
import torch.nn as nn
import torchvision.transforms as T
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader

# Define a tiny CNN manually

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2)
        self.fc   = nn.Linear(8 * 14 * 14, 10)

    def forward(self, x):
        x = torch.relu(self.conv(x))
        x = self.pool(x)
        x = x.view(x.size(0), -1)
        return self.fc(x)

# Load MNIST

train = MNIST(root='./data', train=True, download=True, transform=T.ToTensor())
loader = DataLoader(train, batch_size=64, shuffle=True)

# Train using the repo's helper

net = SimpleCNN().to('cuda' if torch.cuda.is_available() else 'cpu')
from lessons.4-ComputerVision.07-ConvNets.pytorchcv import train
history = train(net, loader, loader, epochs=5)

```

### Equivalent Architecture in Keras

The same network architecture using high-level Keras abstractions:

```python
import tensorflow as tf
from tensorflow.keras import layers, models

def build_cnn():
    model = models.Sequential([
        layers.Conv2D(8, (3,3), padding='same', activation='relu', input_shape=(28,28,1)),
        layers.MaxPooling2D(2),
        layers.Flatten(),
        layers.Dense(10, activation='softmax')
    ])
    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    return model

# Load MNIST via tf.keras.datasets

(train_images, train_labels), _ = tf.keras.datasets.mnist.load_data()
train_images = train_images[..., tf.newaxis]/255.0

model = build_cnn()
model.fit(train_images, train_labels, epochs=5, batch_size=64)

```

### Visualizing Convolution Filters

Both implementations provide `plot_convolution` functions to inspect how filters respond to input data:

```python
import numpy as np
import matplotlib.pyplot as plt
from lessons.4-ComputerVision.07-ConvNets.tfcv import plot_convolution

# Random filter

filter_weights = np.random.randn(3,3)

# Sample image batch (here we use the first 5 MNIST images)

sample_imgs = np.stack([data_train[i][0].numpy().squeeze() for i in range(5)], axis=0)

plot_convolution(sample_imgs, filter_weights, title='Random 3×3 Filter')

```

## Performance and Flexibility Trade-offs

### When to Use From-Scratch Implementation

Choose PyTorch when you need:

- Full control over forward and backward passes
- Custom layer behaviors not covered by standard Keras layers (e.g., sparse convolutions or custom padding strategies)
- Educational clarity regarding convolution mathematics

### When to Use High-Level Frameworks

Opt for Keras/TensorFlow when you require:

- Rapid prototyping with minimal boilerplate
- Seamless deployment to TensorFlow Serving, TensorFlow Lite, or TensorFlow.js
- Graph optimization and XLA compilation for production performance

## Key Implementation Files in the Repository

The microsoft/AI-For-Beginners repository provides parallel implementations that illustrate these concepts:

- **[`lessons/4-ComputerVision/07-ConvNets/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/pytorchcv.py)** — PyTorch utilities for MNIST loading, manual training loops, and convolution visualization via `plot_convolution`
- **[`lessons/4-ComputerVision/07-ConvNets/tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/tfcv.py)** — TensorFlow/Keras helpers including `plot_convolution` for filter visualization and dataset utilities
- **[`lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py)** — Demonstrates extending manual CNNs for transfer learning
- **[`lessons/4-ComputerVision/08-TransferLearning/tfcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/tfcv.py)** — Keras equivalent for transfer learning workflows

## Summary

- Implementing CNN architectures from scratch in PyTorch requires explicit layer definition in `nn.Conv2d` and manual training loops, offering complete control over tensor operations
- Keras/TensorFlow abstracts convolution operations through `keras.layers.Conv2D` and automates training via `model.fit`, reducing boilerplate significantly
- Both approaches support visualization of filter responses through `plot_convolution` functions in their respective utility files
- Choose PyTorch for research requiring custom operations and educational transparency; choose Keras for rapid prototyping and production deployment

## Frequently Asked Questions

### What are the main advantages of implementing CNNs from scratch versus using Keras?

Building from scratch provides complete visibility into tensor operations and gradient flow, essential for research requiring non-standard architectures. Keras accelerates development by automating training loops and weight management, making it ideal for standard models and production deployment.

### Can I access and modify weights in Keras like I do in PyTorch?

Yes, though the workflow differs. In PyTorch, you directly copy tensors using `c.weight.copy_(t)`, while in Keras you access weights through `layer.weights` or `layer.get_weights()` and `layer.set_weights()`, allowing modification but encouraging framework-managed initialization.

### Which approach is better for learning how CNNs work internally?

The from-scratch PyTorch implementation in [`pytorchcv.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/pytorchcv.py) is superior for educational purposes because it requires manual construction of the forward pass and explicit handling of convolution kernels, demystifying the underlying mathematical operations.

### How does performance compare between manual PyTorch implementations and Keras models?

PyTorch's eager execution offers intuitive debugging but can be slower with unvectorized loops. Keras/TensorFlow benefits from graph optimization and XLA compilation, potentially delivering faster inference for large-scale models deployed to production environments.