How to Perform Semantic Segmentation with U-Net: A Complete PyTorch and TensorFlow Guide

Semantic segmentation with U-Net involves encoding an image through convolutional layers to capture high-level context, then decoding it to restore spatial resolution while using skip connections to preserve fine-grained details, and the Microsoft AI-For-Beginners repository provides ready-to-run Jupyter notebooks at lessons/4-ComputerVision/12-Segmentation/ that implement this end-to-end pipeline for both frameworks.

The U-Net architecture, introduced by Ronneberger et al. in U-Net: Convolutional Networks for Biomedical Image Segmentation, remains the gold standard for pixel-level classification tasks. This guide demonstrates how to perform semantic segmentation with U-Net using the complete educational implementation found in the microsoft/AI-For-Beginners repository, which includes functional PyTorch and TensorFlow code designed specifically for beginners.

Understanding the U-Net Architecture

U-Net solves semantic segmentation by combining two complementary pathways. The contracting encoder captures context through successive pooling operations that reduce spatial dimensions while increasing feature depth. The expanding decoder restores the original image resolution through upsampling, but critical spatial information lost during pooling is recovered via skip connections that concatenate encoder features with decoder features at corresponding resolutions.

This symmetric architecture ensures that the final output maintains the same spatial dimensions as the input while assigning a class label to every pixel.

Repository Structure and Key Files

The segmentation lesson in the AI-For-Beginners curriculum provides everything needed to train and evaluate U-Net models:

• lessons/4-ComputerVision/12-Segmentation/README.md – Theoretical overview of semantic segmentation concepts and U-Net mechanics
• lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationPytorch.ipynb – Complete PyTorch implementation including data loading, model definition, and training loops
• lessons/4-ComputerVision/12-Segmentation/SemanticSegmentationTF.ipynb – Equivalent TensorFlow/Keras implementation for TF 2.x users
• environment.yml – Conda environment specification listing required dependencies including PyTorch, TensorFlow, torchvision, and matplotlib

Step-by-Step Implementation Pipeline

Both notebooks follow an identical logical structure to ensure framework-agnostic understanding.

Step 1: Data Loading and Preprocessing

The implementation begins by loading a segmentation dataset (typically Oxford-Pets or synthetic binary masks). Images and their corresponding ground-truth masks are resized to a uniform shape—commonly 128×128 or 256×256 pixels—to enable batch processing.

Step 2: Building the U-Net Model

As implemented in SemanticSegmentationPytorch.ipynb, the model consists of modular ConvBlock components containing two 3×3 convolutional layers with BatchNorm and ReLU activation:

import torch
import torch.nn as nn
import torch.nn.functional as F

class ConvBlock(nn.Module):
    """Two 3×3 conv layers with ReLU and BatchNorm."""
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True),
        )
    def forward(self, x):
        return self.conv(x)

The complete UNet class implements the full encoder-decoder pathway:

class UNet(nn.Module):
    def __init__(self, n_classes=1, in_channels=3, base_ch=64):
        super().__init__()
        # Encoder path

        self.enc1 = ConvBlock(in_channels, base_ch)
        self.enc2 = ConvBlock(base_ch, base_ch*2)
        self.enc3 = ConvBlock(base_ch*2, base_ch*4)
        self.enc4 = ConvBlock(base_ch*4, base_ch*8)
        # Bottleneck

        self.bottleneck = ConvBlock(base_ch*8, base_ch*16)
        # Decoder path with transpose convolutions

        self.up4 = nn.ConvTranspose2d(base_ch*16, base_ch*8, kernel_size=2, stride=2)
        self.dec4 = ConvBlock(base_ch*16, base_ch*8)
        self.up3 = nn.ConvTranspose2d(base_ch*8, base_ch*4, kernel_size=2, stride=2)
        self.dec3 = ConvBlock(base_ch*8, base_ch*4)
        self.up2 = nn.ConvTranspose2d(base_ch*4, base_ch*2, kernel_size=2, stride=2)
        self.dec2 = ConvBlock(base_ch*4, base_ch*2)
        self.up1 = nn.ConvTranspose2d(base_ch*2, base_ch, kernel_size=2, stride=2)
        self.dec1 = ConvBlock(base_ch*2, base_ch)
        # Final 1×1 convolution to class maps

        self.out_conv = nn.Conv2d(base_ch, n_classes, kernel_size=1)

    def forward(self, x):
        # Encoder

        e1 = self.enc1(x)
        e2 = self.enc2(F.max_pool2d(e1, 2))
        e3 = self.enc3(F.max_pool2d(e2, 2))
        e4 = self.enc4(F.max_pool2d(e3, 2))
        # Bottleneck

        b = self.bottleneck(F.max_pool2d(e4, 2))
        # Decoder with skip connections

        d4 = self.dec4(torch.cat([self.up4(b), e4], dim=1))
        d3 = self.dec3(torch.cat([self.up3(d4), e3], dim=1))
        d2 = self.dec2(torch.cat([self.up2(d3), e2], dim=1))
        d1 = self.dec1(torch.cat([self.up1(d2), e1], dim=1))
        return torch.sigmoid(self.out_conv(d1))

Step 3: Loss Functions and Metrics

Semantic segmentation with U-Net typically uses binary cross-entropy for single-class masks or categorical cross-entropy for multi-class segmentation. To handle class imbalance common in segmentation tasks (where foreground pixels are rare compared to background), the implementation includes the Dice coefficient or Intersection over Union (IoU) as additional metrics.

Step 4: Training Loop

The training procedure follows standard supervised learning with data augmentation techniques including random horizontal flips and rotations to improve generalization. The notebooks demonstrate proper handling of batch dimensions and device placement for GPU acceleration.

Step 5: Inference and Visualization

After training, the model predicts segmentation masks on validation images. The notebooks overlay these predictions on original images to visually assess boundary accuracy and class separation.

TensorFlow Implementation Alternative

Users preferring TensorFlow can reference SemanticSegmentationTF.ipynb, which implements the identical U-Net architecture using Keras layers. The encoder uses Conv2D and MaxPooling2D layers, while the decoder employs Conv2DTranspose for upsampling, maintaining the same skip connection architecture as the PyTorch version.

Advanced: Swapping the Encoder Backbone

The repository demonstrates how to replace the standard convolutional encoder with more powerful pretrained backbones such as ResNet-50. This involves loading ImageNet weights, extracting intermediate feature maps at multiple resolutions, and adjusting the channel dimensions in the decoder to match the backbone's output filters.

Summary

• Semantic segmentation with U-Net requires an encoder-decoder architecture with skip connections to preserve spatial fidelity during downsampling and upsampling operations.
• The microsoft/AI-For-Beginners repository provides complete, beginner-friendly implementations in both PyTorch (SemanticSegmentationPytorch.ipynb) and TensorFlow (SemanticSegmentationTF.ipynb).
• Key components include ConvBlock modules for feature extraction, transpose convolutions for upsampling, and concatenation operations for skip connections.
• Training requires appropriate loss functions (cross-entropy) combined with region-based metrics (Dice/IoU) to handle pixel class imbalance.
• The architecture supports encoder swapping for transfer learning, allowing ResNet-50 or similar backbones to replace the standard convolutional stack.

Frequently Asked Questions

What is the difference between semantic segmentation and instance segmentation?

Semantic segmentation assigns a class label to every pixel in an image without distinguishing between separate objects of the same class. Instance segmentation extends this by additionally separating individual object instances, essentially combining semantic segmentation with object detection. The U-Net implementation in the AI-For-Beginners repository focuses on semantic segmentation, producing a single mask per class.

Why are skip connections critical in the U-Net architecture?

Skip connections concatenate high-resolution feature maps from the encoder with upsampled features in the decoder. According to the source code in SemanticSegmentationPytorch.ipynb, these connections occur at each resolution level (e.g., concatenating e4 with the upsampled bottleneck). Without skip connections, the decoder would lose fine spatial details during the encoding process, resulting in blurry boundaries in the final segmentation mask.

Can I use pretrained weights with the U-Net implementation in AI-For-Beginners?

Yes. The notebooks demonstrate how to swap the standard encoder for pretrained backbones like ResNet-50 by loading ImageNet weights and adjusting the channel dimensions that flow into the decoder. This transfer learning approach significantly improves convergence speed and final accuracy, especially when working with limited training data.

What image resolution should I use for U-Net semantic segmentation?

The repository examples typically resize images to 128×128 or 256×256 pixels. Higher resolutions preserve more spatial detail but require exponentially more GPU memory due to the skip connections maintaining full-resolution feature maps. For beginners, starting with 128×128 allows reasonable training times while still demonstrating the U-Net architecture's effectiveness.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}