# How to Implement Object Detection Using TensorFlow in Lesson 11: A Complete RetinaNet Guide

> Implement object detection with TensorFlow in Lesson 11 using RetinaNet. This guide details a ResNet-50 backbone, FPN, and Focal Loss for superior accuracy. Perfect for beginners.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-18

---

**The Microsoft AI-For-Beginners Lesson 11 demonstrates object detection using TensorFlow by implementing RetinaNet, a single-stage detector that combines a ResNet-50 backbone with a Feature Pyramid Network and Focal Loss for handling class imbalance.**

This lesson in the Computer Vision module teaches you to build, train, and deploy a production-ready object detection pipeline. The [`ObjectDetection.ipynb`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) notebook provides the complete implementation, drawing from the official Keras RetinaNet tutorial and adapted for educational clarity.

## Core RetinaNet Architecture in Lesson 11

RetinaNet solves the speed-accuracy trade-off in object detection through three key innovations visible in the source implementation.

### Backbone Network and Feature Pyramid

The model uses **ResNet-50** as its backbone, extracting hierarchical features from input images. The Feature Pyramid Network (FPN) then constructs multi-scale feature maps by combining low-resolution semantic features with high-resolution spatial features.

```python
import tensorflow as tf
from tensorflow.keras import layers, models

# Load ResNet-50 backbone (pre-trained on ImageNet)

backbone = tf.keras.applications.ResNet50(
    include_top=False,
    input_shape=(512, 512, 3),
    weights='imagenet'
)

# Extract intermediate C3, C4, C5 feature maps

c3 = backbone.get_layer('conv3_block4_out').output  # 64x64

c4 = backbone.get_layer('conv4_block6_out').output  # 32x32

c5 = backbone.get_layer('conv5_block3_out').output  # 16x16

```

### Feature Pyramid Network Construction

The FPN generates pyramid levels P3 through P7, enabling detection at multiple scales:

```python
def build_fpn(c3, c4, c5):
    """Build Feature Pyramid Network from backbone outputs."""
    # Top-down pathway with lateral connections

    p5 = layers.Conv2D(256, 1, name='fpn_c5p5')(c5)
    
    p4 = layers.Add(name='fpn_p4add')([
        layers.UpSampling2D(size=(2, 2), name='fpn_p5upsampled')(p5),
        layers.Conv2D(256, 1, name='fpn_c4p4')(c4)
    ])
    p4 = layers.Conv2D(256, 3, padding='same', name='fpn_p4')(p4)
    
    p3 = layers.Add(name='fpn_p3add')([
        layers.UpSampling2D(size=(2, 2), name='fpn_p4upsampled')(p4),
        layers.Conv2D(256, 1, name='fpn_c3p3')(c3)
    ])
    p3 = layers.Conv2D(256, 3, padding='same', name='fpn_p3')(p3)
    
    # Additional pyramid levels for large objects

    p6 = layers.Conv2D(256, 3, strides=2, padding='same', name='fpn_p6')(c5)
    p7 = layers.Conv2D(256, 3, strides=2, padding='same', activation='relu', name='fpn_p7')(p6)
    
    return [p3, p4, p5, p6, p7]

```

## Detection Heads: Classification and Regression

RetinaNet uses two parallel sub-networks operating on each FPN level.

### Classification Sub-Network

```python
def build_classification_head(num_classes, num_anchors=9):
    """Predict object class probabilities for each anchor."""
    def head(input_feature):
        x = input_feature
        # Four stacked 3x3 conv layers

        for i in range(4):
            x = layers.Conv2D(
                256, 3, padding='same', activation='relu',
                name=f'cls_conv_{i}'
            )(x)
        # Output: (H, W, num_anchors * num_classes)

        x = layers.Conv2D(
            num_anchors * num_classes, 3, padding='same',
            name='cls_output'
        )(x)
        # Reshape to (num_anchors, num_classes)

        x = layers.Reshape((-1, num_classes))(x)
        return x
    return head

```

### Box Regression Sub-Network

```python
def build_regression_head(num_anchors=9):
    """Predict bounding box coordinate adjustments."""
    def head(input_feature):
        x = input_feature
        for i in range(4):
            x = layers.Conv2D(
                256, 3, padding='same', activation='relu',
                name=f'box_conv_{i}'
            )(x)
        # Output: (H, W, num_anchors * 4) for (dx, dy, dw, dh)

        x = layers.Conv2D(
            num_anchors * 4, 3, padding='same',
            name='box_output'
        )(x)
        x = layers.Reshape((-1, 4))(x)
        return x
    return head

```

## Focal Loss: Handling Foreground-Background Imbalance

The **Focal Loss** function is critical to RetinaNet's success, down-weighting easy negatives so training focuses on hard examples.

```python
def focal_loss(gamma=2.0, alpha=0.25):
    """
    Focal Loss for addressing class imbalance in detection.
    
    gamma: focusing parameter (higher = more focus on hard examples)
    alpha: weighting factor for positive class
    """
    def loss_fn(y_true, y_pred):
        # Binary cross-entropy

        bce = tf.keras.losses.binary_crossentropy(y_true, y_pred, from_logits=True)
        
        # Apply sigmoid to get probabilities

        p = tf.nn.sigmoid(y_pred)
        
        # Modulating factor: (1 - p)^gamma for positive, p^gamma for negative

        p_t = (y_true * p) + ((1 - y_true) * (1 - p))
        alpha_t = y_true * alpha + (1 - y_true) * (1 - alpha)
        
        focal_weight = alpha_t * tf.pow(1.0 - p_t, gamma)
        
        return tf.reduce_sum(focal_weight * bce, axis=-1)
    
    return loss_fn

```

## Complete Model Assembly

```python
def build_retinanet(num_classes, num_anchors=9):
    """Assemble complete RetinaNet model."""
    # Input and backbone

    input_layer = layers.Input(shape=(512, 512, 3), name='image_input')
    backbone = tf.keras.applications.ResNet50(
        include_top=False, weights='imagenet'
    )
    c3_output = backbone.get_layer('conv3_block4_out').output
    c4_output = backbone.get_layer('conv4_block6_out').output
    c5_output = backbone.get_layer('conv5_block3_out').output
    
    # FPN

    p3, p4, p5, p6, p7 = build_fpn(c3_output, c4_output, c5_output)
    pyramid_features = [p3, p4, p5, p6, p7]
    
    # Shared heads applied to each pyramid level

    cls_head_fn = build_classification_head(num_classes, num_anchors)
    box_head_fn = build_regression_head(num_anchors)
    
    cls_outputs = [cls_head_fn(f) for f in pyramid_features]
    box_outputs = [box_head_fn(f) for f in pyramid_features]
    
    # Concatenate predictions from all levels

    classifications = layers.Concatenate(axis=1, name='classifications')(cls_outputs)
    box_regressions = layers.Concatenate(axis=1, name='box_regressions')(box_outputs)
    
    return models.Model(
        inputs=input_layer,
        outputs=[classifications, box_regressions],
        name='RetinaNet'
    )

```

## Training Pipeline with TensorFlow Datasets

The notebook demonstrates loading data via **TensorFlow Datasets** and custom preprocessing.

```python
import tensorflow_datasets as tfds

def load_detection_dataset(dataset_name='voc/2007', batch_size=8):
    """Load and preprocess detection dataset."""
    
    (train_ds, val_ds), info = tfds.load(
        dataset_name,
        split=['train', 'validation'],
        with_info=True,
        as_supervised=False  # Returns dict with 'image', 'objects', etc.

    )
    
    def preprocess(example):
        image = tf.cast(example['image'], tf.float32)
        image = tf.image.resize(image, (512, 512))
        image = image / 255.0
        
        # Extract boxes and labels from nested structure

        boxes = example['objects']['bbox']  # [N, 4] in normalized [y_min, x_min, y_max, x_max]

        labels = example['objects']['label']  # [N]

        
        return image, {'boxes': boxes, 'labels': labels}
    
    train_ds = train_ds.map(preprocess).batch(batch_size).prefetch(tf.data.AUTOTUNE)
    val_ds = val_ds.map(preprocess).batch(batch_size).prefetch(tf.data.AUTOTUNE)
    
    return train_ds, val_ds, info

# Build and compile model

num_classes = 20  # VOC has 20 classes

model = build_retinanet(num_classes=num_classes)

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
    loss=[focal_loss(gamma=2.0, alpha=0.25), 
          tf.keras.losses.Huber(delta=0.1)],  # Smooth L1 for regression

    loss_weights=[1.0, 1.0]
)

# Train with tensorboard logging

train_ds, val_ds, info = load_detection_dataset()
model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=50,
    callbacks=[
        tf.keras.callbacks.TensorBoard(log_dir='./logs'),
        tf.keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)
    ]
)

```

## Inference and Non-Maximum Suppression

Post-processing converts network outputs to final detections.

```python
def decode_predictions(classifications, box_regressions, anchors,
                       score_threshold=0.5, nms_iou_threshold=0.5,
                       max_detections=100):
    """
    Decode network outputs to final bounding boxes.
    
    classifications: [batch, num_anchors, num_classes]
    box_regressions: [batch, num_anchors, 4] as (dx, dy, dw, dh)
    anchors: [num_anchors, 4] as (x_center, y_center, width, height)
    """
    # Apply sigmoid to get class probabilities

    scores = tf.nn.sigmoid(classifications)
    
    # Transform anchor-relative deltas to absolute box coordinates

    anchor_centers_x = anchors[:, 0]
    anchor_centers_y = anchors[:, 1]
    anchor_widths = anchors[:, 2]
    anchor_heights = anchors[:, 3]
    
    # dx, dy are relative to anchor size; dw, dh are log-space

    pred_ctr_x = box_regressions[..., 0] * anchor_widths + anchor_centers_x
    pred_ctr_y = box_regressions[..., 1] * anchor_heights + anchor_centers_y
    pred_w = tf.exp(box_regressions[..., 2]) * anchor_widths
    pred_h = tf.exp(box_regressions[..., 3]) * anchor_heights
    
    # Convert to [y_min, x_min, y_max, x_max] format

    y_min = pred_ctr_y - pred_h / 2.0
    x_min = pred_ctr_x - pred_w / 2.0
    y_max = pred_ctr_y + pred_h / 2.0
    x_max = pred_ctr_x + pred_w / 2.0
    boxes = tf.stack([y_min, x_min, y_max, x_max], axis=-1)
    
    # Apply NMS per class

    detections = tf.image.combined_non_max_suppression(
        boxes=tf.expand_dims(boxes, axis=2),  # [batch, num_boxes, 1, 4]

        scores=scores,
        max_output_size_per_class=max_detections,
        max_total_size=max_detections,
        iou_threshold=nms_iou_threshold,
        score_threshold=score_threshold
    )
    
    return detections  # Returns (nmsed_boxes, nmsed_scores, nmsed_classes, valid_detections)

# Example inference

image_batch = tf.random.normal((1, 512, 512, 3))
cls_output, box_output = model(image_batch, training=False)

# anchors would be pre-computed based on image size and pyramid levels

detections = decode_predictions(cls_output, box_output, anchors)

```

## Key Files and Resources

| File | Description | Direct Link |
|------|-------------|-------------|
| `ObjectDetection.ipynb` | Primary lesson notebook with complete TensorFlow implementation | [`lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) |
| Keras RetinaNet Tutorial | Official reference implementation and architectural details | [keras.io/examples/vision/retinanet/](https://keras.io/examples/vision/retinanet/) |
| keras-retinanet library | Production-ready implementation with pre-trained weights | [github.com/fizyr/keras-retinanet](https://github.com/fizyr/keras-retinanet) |

## Summary

- **RetinaNet** combines a **ResNet-50 backbone** with a **Feature Pyramid Network** to detect objects at multiple scales efficiently.
- **Focal Loss** solves the extreme foreground-background imbalance that plagues single-stage detectors, making RetinaNet competitive with two-stage methods like Faster R-CNN.
- The **classification and regression heads** share architecture but operate independently on each FPN level, with predictions concatenated across all scales.
- **Anchor-based detection** uses pre-defined boxes at multiple aspect ratios and scales; the network predicts deltas to refine these anchors.
- The [`ObjectDetection.ipynb`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb) notebook provides a complete, runnable pipeline from data loading through training to inference with NMS.

## Frequently Asked Questions

### What makes RetinaNet different from YOLO or SSD?

RetinaNet introduces **Focal Loss** to address the class imbalance problem that hurts single-stage detectors. While YOLO and SSD use hard negative mining or complex sampling strategies, RetinaNet's focal loss automatically down-weights easy negatives, allowing the network to focus on hard examples during training. This achieves accuracy comparable to two-stage detectors (like Faster R-CNN) while maintaining the speed advantages of single-stage approaches.

### Why does the Feature Pyramid Network use P3 through P7 levels?

The **P3-P7 pyramid levels** correspond to different receptive field sizes, enabling detection across a wide range of object scales. **P3** (stride 8, largest spatial resolution) captures small objects, while **P7** (stride 128) handles large objects. Each level shares the same head architecture, making the model scale-invariant without requiring image pyramids at inference time.

### How do I generate anchor boxes for my custom image size?

Anchor generation follows a systematic grid pattern: for each FPN level at stride **s**, anchors are placed at positions `(s/2 + i*s, s/2 + j*s)`. Each position uses **9 anchors** (3 scales × 3 aspect ratios: 1:1, 1:2, 2:1). For a 512×512 input, this yields approximately 60K-80K anchors across all levels. The [`keras-retinanet`](https://github.com/fizyr/keras-retinanet) library provides optimized anchor generation utilities in [`keras_retinanet/utils/anchors.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/keras_retinanet/utils/anchors.py).