How to Implement Object Detection Using TensorFlow in Lesson 11: A Complete RetinaNet Guide

The Microsoft AI-For-Beginners Lesson 11 demonstrates object detection using TensorFlow by implementing RetinaNet, a single-stage detector that combines a ResNet-50 backbone with a Feature Pyramid Network and Focal Loss for handling class imbalance.

This lesson in the Computer Vision module teaches you to build, train, and deploy a production-ready object detection pipeline. The ObjectDetection.ipynb notebook provides the complete implementation, drawing from the official Keras RetinaNet tutorial and adapted for educational clarity.

Core RetinaNet Architecture in Lesson 11

RetinaNet solves the speed-accuracy trade-off in object detection through three key innovations visible in the source implementation.

Backbone Network and Feature Pyramid

The model uses ResNet-50 as its backbone, extracting hierarchical features from input images. The Feature Pyramid Network (FPN) then constructs multi-scale feature maps by combining low-resolution semantic features with high-resolution spatial features.

import tensorflow as tf
from tensorflow.keras import layers, models

# Load ResNet-50 backbone (pre-trained on ImageNet)

backbone = tf.keras.applications.ResNet50(
    include_top=False,
    input_shape=(512, 512, 3),
    weights='imagenet'
)

# Extract intermediate C3, C4, C5 feature maps

c3 = backbone.get_layer('conv3_block4_out').output  # 64x64

c4 = backbone.get_layer('conv4_block6_out').output  # 32x32

c5 = backbone.get_layer('conv5_block3_out').output  # 16x16

Feature Pyramid Network Construction

The FPN generates pyramid levels P3 through P7, enabling detection at multiple scales:

def build_fpn(c3, c4, c5):
    """Build Feature Pyramid Network from backbone outputs."""
    # Top-down pathway with lateral connections

    p5 = layers.Conv2D(256, 1, name='fpn_c5p5')(c5)
    
    p4 = layers.Add(name='fpn_p4add')([
        layers.UpSampling2D(size=(2, 2), name='fpn_p5upsampled')(p5),
        layers.Conv2D(256, 1, name='fpn_c4p4')(c4)
    ])
    p4 = layers.Conv2D(256, 3, padding='same', name='fpn_p4')(p4)
    
    p3 = layers.Add(name='fpn_p3add')([
        layers.UpSampling2D(size=(2, 2), name='fpn_p4upsampled')(p4),
        layers.Conv2D(256, 1, name='fpn_c3p3')(c3)
    ])
    p3 = layers.Conv2D(256, 3, padding='same', name='fpn_p3')(p3)
    
    # Additional pyramid levels for large objects

    p6 = layers.Conv2D(256, 3, strides=2, padding='same', name='fpn_p6')(c5)
    p7 = layers.Conv2D(256, 3, strides=2, padding='same', activation='relu', name='fpn_p7')(p6)
    
    return [p3, p4, p5, p6, p7]

Detection Heads: Classification and Regression

RetinaNet uses two parallel sub-networks operating on each FPN level.

Classification Sub-Network

def build_classification_head(num_classes, num_anchors=9):
    """Predict object class probabilities for each anchor."""
    def head(input_feature):
        x = input_feature
        # Four stacked 3x3 conv layers

        for i in range(4):
            x = layers.Conv2D(
                256, 3, padding='same', activation='relu',
                name=f'cls_conv_{i}'
            )(x)
        # Output: (H, W, num_anchors * num_classes)

        x = layers.Conv2D(
            num_anchors * num_classes, 3, padding='same',
            name='cls_output'
        )(x)
        # Reshape to (num_anchors, num_classes)

        x = layers.Reshape((-1, num_classes))(x)
        return x
    return head

Box Regression Sub-Network

def build_regression_head(num_anchors=9):
    """Predict bounding box coordinate adjustments."""
    def head(input_feature):
        x = input_feature
        for i in range(4):
            x = layers.Conv2D(
                256, 3, padding='same', activation='relu',
                name=f'box_conv_{i}'
            )(x)
        # Output: (H, W, num_anchors * 4) for (dx, dy, dw, dh)

        x = layers.Conv2D(
            num_anchors * 4, 3, padding='same',
            name='box_output'
        )(x)
        x = layers.Reshape((-1, 4))(x)
        return x
    return head

Focal Loss: Handling Foreground-Background Imbalance

The Focal Loss function is critical to RetinaNet's success, down-weighting easy negatives so training focuses on hard examples.

def focal_loss(gamma=2.0, alpha=0.25):
    """
    Focal Loss for addressing class imbalance in detection.
    
    gamma: focusing parameter (higher = more focus on hard examples)
    alpha: weighting factor for positive class
    """
    def loss_fn(y_true, y_pred):
        # Binary cross-entropy

        bce = tf.keras.losses.binary_crossentropy(y_true, y_pred, from_logits=True)
        
        # Apply sigmoid to get probabilities

        p = tf.nn.sigmoid(y_pred)
        
        # Modulating factor: (1 - p)^gamma for positive, p^gamma for negative

        p_t = (y_true * p) + ((1 - y_true) * (1 - p))
        alpha_t = y_true * alpha + (1 - y_true) * (1 - alpha)
        
        focal_weight = alpha_t * tf.pow(1.0 - p_t, gamma)
        
        return tf.reduce_sum(focal_weight * bce, axis=-1)
    
    return loss_fn

Complete Model Assembly

def build_retinanet(num_classes, num_anchors=9):
    """Assemble complete RetinaNet model."""
    # Input and backbone

    input_layer = layers.Input(shape=(512, 512, 3), name='image_input')
    backbone = tf.keras.applications.ResNet50(
        include_top=False, weights='imagenet'
    )
    c3_output = backbone.get_layer('conv3_block4_out').output
    c4_output = backbone.get_layer('conv4_block6_out').output
    c5_output = backbone.get_layer('conv5_block3_out').output
    
    # FPN

    p3, p4, p5, p6, p7 = build_fpn(c3_output, c4_output, c5_output)
    pyramid_features = [p3, p4, p5, p6, p7]
    
    # Shared heads applied to each pyramid level

    cls_head_fn = build_classification_head(num_classes, num_anchors)
    box_head_fn = build_regression_head(num_anchors)
    
    cls_outputs = [cls_head_fn(f) for f in pyramid_features]
    box_outputs = [box_head_fn(f) for f in pyramid_features]
    
    # Concatenate predictions from all levels

    classifications = layers.Concatenate(axis=1, name='classifications')(cls_outputs)
    box_regressions = layers.Concatenate(axis=1, name='box_regressions')(box_outputs)
    
    return models.Model(
        inputs=input_layer,
        outputs=[classifications, box_regressions],
        name='RetinaNet'
    )

Training Pipeline with TensorFlow Datasets

The notebook demonstrates loading data via TensorFlow Datasets and custom preprocessing.

import tensorflow_datasets as tfds

def load_detection_dataset(dataset_name='voc/2007', batch_size=8):
    """Load and preprocess detection dataset."""
    
    (train_ds, val_ds), info = tfds.load(
        dataset_name,
        split=['train', 'validation'],
        with_info=True,
        as_supervised=False  # Returns dict with 'image', 'objects', etc.

    )
    
    def preprocess(example):
        image = tf.cast(example['image'], tf.float32)
        image = tf.image.resize(image, (512, 512))
        image = image / 255.0
        
        # Extract boxes and labels from nested structure

        boxes = example['objects']['bbox']  # [N, 4] in normalized [y_min, x_min, y_max, x_max]

        labels = example['objects']['label']  # [N]

        
        return image, {'boxes': boxes, 'labels': labels}
    
    train_ds = train_ds.map(preprocess).batch(batch_size).prefetch(tf.data.AUTOTUNE)
    val_ds = val_ds.map(preprocess).batch(batch_size).prefetch(tf.data.AUTOTUNE)
    
    return train_ds, val_ds, info

# Build and compile model

num_classes = 20  # VOC has 20 classes

model = build_retinanet(num_classes=num_classes)

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
    loss=[focal_loss(gamma=2.0, alpha=0.25), 
          tf.keras.losses.Huber(delta=0.1)],  # Smooth L1 for regression

    loss_weights=[1.0, 1.0]
)

# Train with tensorboard logging

train_ds, val_ds, info = load_detection_dataset()
model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=50,
    callbacks=[
        tf.keras.callbacks.TensorBoard(log_dir='./logs'),
        tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)
    ]
)

Inference and Non-Maximum Suppression

Post-processing converts network outputs to final detections.

def decode_predictions(classifications, box_regressions, anchors,
                       score_threshold=0.5, nms_iou_threshold=0.5,
                       max_detections=100):
    """
    Decode network outputs to final bounding boxes.
    
    classifications: [batch, num_anchors, num_classes]
    box_regressions: [batch, num_anchors, 4] as (dx, dy, dw, dh)
    anchors: [num_anchors, 4] as (x_center, y_center, width, height)
    """
    # Apply sigmoid to get class probabilities

    scores = tf.nn.sigmoid(classifications)
    
    # Transform anchor-relative deltas to absolute box coordinates

    anchor_centers_x = anchors[:, 0]
    anchor_centers_y = anchors[:, 1]
    anchor_widths = anchors[:, 2]
    anchor_heights = anchors[:, 3]
    
    # dx, dy are relative to anchor size; dw, dh are log-space

    pred_ctr_x = box_regressions[..., 0] * anchor_widths + anchor_centers_x
    pred_ctr_y = box_regressions[..., 1] * anchor_heights + anchor_centers_y
    pred_w = tf.exp(box_regressions[..., 2]) * anchor_widths
    pred_h = tf.exp(box_regressions[..., 3]) * anchor_heights
    
    # Convert to [y_min, x_min, y_max, x_max] format

    y_min = pred_ctr_y - pred_h / 2.0
    x_min = pred_ctr_x - pred_w / 2.0
    y_max = pred_ctr_y + pred_h / 2.0
    x_max = pred_ctr_x + pred_w / 2.0
    boxes = tf.stack([y_min, x_min, y_max, x_max], axis=-1)
    
    # Apply NMS per class

    detections = tf.image.combined_non_max_suppression(
        boxes=tf.expand_dims(boxes, axis=2),  # [batch, num_boxes, 1, 4]

        scores=scores,
        max_output_size_per_class=max_detections,
        max_total_size=max_detections,
        iou_threshold=nms_iou_threshold,
        score_threshold=score_threshold
    )
    
    return detections  # Returns (nmsed_boxes, nmsed_scores, nmsed_classes, valid_detections)

# Example inference

image_batch = tf.random.normal((1, 512, 512, 3))
cls_output, box_output = model(image_batch, training=False)

# anchors would be pre-computed based on image size and pyramid levels

detections = decode_predictions(cls_output, box_output, anchors)

Key Files and Resources

File	Description	Direct Link
ObjectDetection.ipynb	Primary lesson notebook with complete TensorFlow implementation	lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb
Keras RetinaNet Tutorial	Official reference implementation and architectural details	keras.io/examples/vision/retinanet/
keras-retinanet library	Production-ready implementation with pre-trained weights	github.com/fizyr/keras-retinanet

Summary

• RetinaNet combines a ResNet-50 backbone with a Feature Pyramid Network to detect objects at multiple scales efficiently.
• Focal Loss solves the extreme foreground-background imbalance that plagues single-stage detectors, making RetinaNet competitive with two-stage methods like Faster R-CNN.
• The classification and regression heads share architecture but operate independently on each FPN level, with predictions concatenated across all scales.
• Anchor-based detection uses pre-defined boxes at multiple aspect ratios and scales; the network predicts deltas to refine these anchors.
• The ObjectDetection.ipynb notebook provides a complete, runnable pipeline from data loading through training to inference with NMS.

Frequently Asked Questions

What makes RetinaNet different from YOLO or SSD?

RetinaNet introduces Focal Loss to address the class imbalance problem that hurts single-stage detectors. While YOLO and SSD use hard negative mining or complex sampling strategies, RetinaNet's focal loss automatically down-weights easy negatives, allowing the network to focus on hard examples during training. This achieves accuracy comparable to two-stage detectors (like Faster R-CNN) while maintaining the speed advantages of single-stage approaches.

Why does the Feature Pyramid Network use P3 through P7 levels?

The P3-P7 pyramid levels correspond to different receptive field sizes, enabling detection across a wide range of object scales. P3 (stride 8, largest spatial resolution) captures small objects, while P7 (stride 128) handles large objects. Each level shares the same head architecture, making the model scale-invariant without requiring image pyramids at inference time.

How do I generate anchor boxes for my custom image size?

Anchor generation follows a systematic grid pattern: for each FPN level at stride s, anchors are placed at positions (s/2 + i*s, s/2 + j*s). Each position uses 9 anchors (3 scales × 3 aspect ratios: 1:1, 1:2, 2:1). For a 512×512 input, this yields approximately 60K-80K anchors across all levels. The keras-retinanet library provides optimized anchor generation utilities in keras_retinanet/utils/anchors.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}