Build CNN from Scratch with Keras: AI for Beginners Complete Guide

You can build a convolutional neural network from scratch using TensorFlow Keras by stacking Conv2D, MaxPooling2D, and Dense layers, then training with model.fit() on normalized image data.

The AI for Beginners curriculum by Microsoft provides a beginner-friendly pathway to master computer vision fundamentals. Located in the microsoft/AI-For-Beginners repository, lesson 4 delivers hands-on Jupyter notebooks that teach you how to build CNNs from the ground up using Keras. This guide extracts the exact implementation details from the official ConvNetsTF.ipynb notebook to get you training your first image classifier in minutes.

Understanding the CNN Architecture

A convolutional neural network processes images through hierarchical feature extraction. According to the source code in lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb, the architecture follows a feature extractor → classifier pattern.

Core Components

• Input Layer (tf.keras.layers.Input): Defines the tensor shape, typically (28, 28, 1) for grayscale MNIST images.
• Convolutional Layers (Conv2D): Learn spatial filters using learnable kernels. The notebook uses filters=32, kernel_size=3, activation='relu' to detect edges and textures.
• Pooling Layers (MaxPooling2D): Reduce spatial dimensions with pool_size=2, providing translation invariance and reducing computation.
• Flatten: Converts 3-D feature maps into 1-D vectors before feeding into dense layers.
• Dense Layers: Perform high-level reasoning. The hidden layer uses units=128, activation='relu', while the output layer uses units=10, activation='softmax' for classification.

This stack creates a progressive abstraction: early layers detect edges, middle layers combine them into shapes, and final layers classify the object.

Building Your First Keras CNN

The repository provides a complete, executable implementation in lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb that requires only TensorFlow and NumPy.

Loading and Preprocessing Data

Normalization stabilizes training by scaling pixel values to the [0,1] range.

import tensorflow as tf
import numpy as np

# Load MNIST dataset

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Reshape and normalize

x_train = x_train[..., np.newaxis] / 255.0   # shape: (60000, 28, 28, 1)

x_test = x_test[..., np.newaxis] / 255.0

The np.newaxis operation adds a channel dimension required by Keras convolutional layers.

Defining the Model Architecture

The Sequential API provides a clean, linear stack of layers as implemented in the official notebook:

from tensorflow.keras import layers, models

model = models.Sequential([
    layers.Input(shape=(28, 28, 1)),
    layers.Conv2D(32, 3, activation='relu'),      # Output: (26, 26, 32)

    layers.MaxPooling2D(2),                       # Output: (13, 13, 32)

    layers.Conv2D(64, 3, activation='relu'),      # Output: (11, 11, 64)

    layers.MaxPooling2D(2),                       # Output: (5, 5, 64)

    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])

Key architectural decisions in this code:

• Small kernels (3×3): Capture fine details while keeping parameter counts manageable.
• Doubling filters: The second convolution uses 64 filters to capture more complex patterns than the first 32-filter layer.
• Softmax output: Produces probability distributions across 10 digit classes.

Compiling and Training

The notebook uses Adam optimizer and sparse categorical crossentropy for efficient multi-class classification:

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

history = model.fit(x_train, y_train,
                    epochs=10,
                    batch_size=64,
                    validation_split=0.1,
                    verbose=2)

The validation_split=0.1 parameter automatically reserves 10% of training data for validation, allowing you to monitor overfitting without manual data separation.

Enhancing the Model

The AI for Beginners curriculum suggests several improvements to boost generalization.

Adding Dropout Regularization

Insert Dropout(0.5) between the flatten and dense layers to prevent overfitting:

model = models.Sequential([
    layers.Input(shape=(28, 28, 1)),
    layers.Conv2D(32, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dropout(0.5),                          # Randomly zero 50% of inputs

    layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])

Data Augmentation Pipeline

For more robust training on limited datasets, prepend an augmentation layer as shown in the notebook's experiment sections:

data_augmentation = tf.keras.Sequential([
    layers.RandomFlip('horizontal'),
    layers.RandomRotation(0.1),
])

model = models.Sequential([
    layers.Input(shape=(28, 28, 1)),
    data_augmentation,                          # Applied during training only

    layers.Conv2D(32, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dense(10, activation='softmax')
])

Visualizing Training Progress

Understanding convergence requires plotting the training history. The notebook includes matplotlib visualizations to diagnose underfitting or overfitting:

import matplotlib.pyplot as plt

plt.plot(history.history['accuracy'], label='train')
plt.plot(history.history['val_accuracy'], label='val')
plt.title('Training & Validation Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

A healthy training run shows both curves increasing together. If validation accuracy plateaus while training accuracy rises, the model is overfitting and requires regularization.

Summary

• Clone the repository at microsoft/AI-For-Beginners and open lessons/4-ComputerVision/07-ConvNets/ConvNetsTF.ipynb for the canonical implementation.
• Normalize pixel values by dividing by 255.0 before feeding images to the network.
• Stack Conv2D and MaxPooling2D layers to extract hierarchical features from raw pixels.
• Use the Sequential API for clear, readable model definitions that mirror the mathematical flow.
• Monitor validation metrics during model.fit() to catch overfitting early and adjust hyperparameters.

Frequently Asked Questions

How do I install the required dependencies for the AI for Beginners CNN notebooks?

Install the conda environment defined in environment.yml at the repository root. This file specifies TensorFlow 2.x, PyTorch, Jupyter, and other computer vision libraries required to execute ConvNetsTF.ipynb without import errors.

What is the difference between the Keras and PyTorch implementations in the repository?

The ConvNetsTF.ipynb notebook uses Keras's high-level Sequential API with model.fit() for training, while ConvNetsPyTorch.ipynb implements the same architecture using torch.nn.Module and explicit training loops. Both achieve identical accuracy but Keras abstracts tensor operations for beginners, whereas PyTorch exposes the autograd mechanism.

Why does the CNN use sparse categorical crossentropy instead of categorical crossentropy?

The MNIST labels in y_train are integer indices (0-9) rather than one-hot encoded vectors. sparse_categorical_crossentropy computes the loss directly from integer labels, saving memory compared to converting labels to one-hot format required by standard categorical_crossentropy.

How can I improve the test accuracy beyond the baseline 98%?

Add Dropout layers after flattening to reduce overfitting, implement data augmentation with RandomFlip and RandomRotation to expand your training set artificially, and experiment with EarlyStopping callbacks to halt training when validation loss plateaus. The notebook suggests these as extension exercises for deeper learning.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}