Implementing CNN Architectures from Scratch vs Using Keras/TensorFlow: A Deep Dive

Building CNNs from scratch in PyTorch provides complete control over tensor operations and training loops, while Keras/TensorFlow abstracts boilerplate through high-level APIs like model.fit, making each approach optimal for different stages of the machine learning workflow.

Convolutional Neural Networks (CNNs) form the backbone of modern computer vision, and the microsoft/AI-For-Beginners repository demonstrates two fundamentally different approaches to constructing these architectures. Whether you implement CNN architectures from scratch using PyTorch's explicit layer definitions or leverage the abstraction of Keras/TensorFlow, understanding both paradigms is essential for selecting the right tool for research, education, or production deployment.

Layer Definition and Weight Control

Manual Implementation in PyTorch

In lessons/4-ComputerVision/07-ConvNets/pytorchcv.py, the repository demonstrates explicit instantiation of convolutional layers using nn.Conv2d, where developers manually initialize weights and handle the forward pass. This approach exposes the underlying mathematics—you directly manipulate weight tensors through operations like c.weight.copy_(t), granting full control over initialization schemes, regularization, and custom constraints.

High-Level Abstraction in Keras/TensorFlow

Conversely, the TensorFlow implementation in tfcv.py utilizes tf.nn.conv2d or keras.layers.Conv2D, where the framework manages kernel tensors internally. While weights remain accessible via layer.weights, the API encourages allowing Keras to handle the weight lifecycle, significantly reducing code complexity for standard architectures.

Training Loop Implementation

Explicit Training Logic

The PyTorch approach requires manual construction of training logic through functions like train_epoch, validate, and train as seen in pytorchcv.py. Developers must explicitly iterate over batches, compute gradients with an optimizer, calculate loss, and track accuracy metrics—providing transparency into every optimization step.

Automated Training with model.fit

Keras eliminates this boilerplate through model.fit, which automatically handles batch iteration, loss computation, and metric updates. This abstraction, demonstrated in the repository's examples, dramatically accelerates experimentation cycles while seamlessly integrating with the broader TensorFlow ecosystem including tf.data pipelines and TensorBoard monitoring.

Practical Code Examples

Building a CNN from Scratch in PyTorch

The following implementation from the repository shows explicit layer definition and manual training:

import torch
import torch.nn as nn
import torchvision.transforms as T
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader

# Define a tiny CNN manually

class SimpleCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2)
        self.fc   = nn.Linear(8 * 14 * 14, 10)

    def forward(self, x):
        x = torch.relu(self.conv(x))
        x = self.pool(x)
        x = x.view(x.size(0), -1)
        return self.fc(x)

# Load MNIST

train = MNIST(root='./data', train=True, download=True, transform=T.ToTensor())
loader = DataLoader(train, batch_size=64, shuffle=True)

# Train using the repo's helper

net = SimpleCNN().to('cuda' if torch.cuda.is_available() else 'cpu')
from lessons.4-ComputerVision.07-ConvNets.pytorchcv import train
history = train(net, loader, loader, epochs=5)

Equivalent Architecture in Keras

The same network architecture using high-level Keras abstractions:

import tensorflow as tf
from tensorflow.keras import layers, models

def build_cnn():
    model = models.Sequential([
        layers.Conv2D(8, (3,3), padding='same', activation='relu', input_shape=(28,28,1)),
        layers.MaxPooling2D(2),
        layers.Flatten(),
        layers.Dense(10, activation='softmax')
    ])
    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    return model

# Load MNIST via tf.keras.datasets

(train_images, train_labels), _ = tf.keras.datasets.mnist.load_data()
train_images = train_images[..., tf.newaxis]/255.0

model = build_cnn()
model.fit(train_images, train_labels, epochs=5, batch_size=64)

Visualizing Convolution Filters

Both implementations provide plot_convolution functions to inspect how filters respond to input data:

import numpy as np
import matplotlib.pyplot as plt
from lessons.4-ComputerVision.07-ConvNets.tfcv import plot_convolution

# Random filter

filter_weights = np.random.randn(3,3)

# Sample image batch (here we use the first 5 MNIST images)

sample_imgs = np.stack([data_train[i][0].numpy().squeeze() for i in range(5)], axis=0)

plot_convolution(sample_imgs, filter_weights, title='Random 3×3 Filter')

Performance and Flexibility Trade-offs

When to Use From-Scratch Implementation

Choose PyTorch when you need:

• Full control over forward and backward passes
• Custom layer behaviors not covered by standard Keras layers (e.g., sparse convolutions or custom padding strategies)
• Educational clarity regarding convolution mathematics

When to Use High-Level Frameworks

Opt for Keras/TensorFlow when you require:

• Rapid prototyping with minimal boilerplate
• Seamless deployment to TensorFlow Serving, TensorFlow Lite, or TensorFlow.js
• Graph optimization and XLA compilation for production performance

Key Implementation Files in the Repository

The microsoft/AI-For-Beginners repository provides parallel implementations that illustrate these concepts:

• lessons/4-ComputerVision/07-ConvNets/pytorchcv.py — PyTorch utilities for MNIST loading, manual training loops, and convolution visualization via plot_convolution
• lessons/4-ComputerVision/07-ConvNets/tfcv.py — TensorFlow/Keras helpers including plot_convolution for filter visualization and dataset utilities
• lessons/4-ComputerVision/08-TransferLearning/pytorchcv.py — Demonstrates extending manual CNNs for transfer learning
• lessons/4-ComputerVision/08-TransferLearning/tfcv.py — Keras equivalent for transfer learning workflows

Summary

• Implementing CNN architectures from scratch in PyTorch requires explicit layer definition in nn.Conv2d and manual training loops, offering complete control over tensor operations
• Keras/TensorFlow abstracts convolution operations through keras.layers.Conv2D and automates training via model.fit, reducing boilerplate significantly
• Both approaches support visualization of filter responses through plot_convolution functions in their respective utility files
• Choose PyTorch for research requiring custom operations and educational transparency; choose Keras for rapid prototyping and production deployment

Frequently Asked Questions

What are the main advantages of implementing CNNs from scratch versus using Keras?

Building from scratch provides complete visibility into tensor operations and gradient flow, essential for research requiring non-standard architectures. Keras accelerates development by automating training loops and weight management, making it ideal for standard models and production deployment.

Can I access and modify weights in Keras like I do in PyTorch?

Yes, though the workflow differs. In PyTorch, you directly copy tensors using c.weight.copy_(t), while in Keras you access weights through layer.weights or layer.get_weights() and layer.set_weights(), allowing modification but encouraging framework-managed initialization.

Which approach is better for learning how CNNs work internally?

The from-scratch PyTorch implementation in pytorchcv.py is superior for educational purposes because it requires manual construction of the forward pass and explicit handling of convolution kernels, demystifying the underlying mathematical operations.

How does performance compare between manual PyTorch implementations and Keras models?

PyTorch's eager execution offers intuitive debugging but can be slower with unvectorized loops. Keras/TensorFlow benefits from graph optimization and XLA compilation, potentially delivering faster inference for large-scale models deployed to production environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}