# How to Implement Object Detection Using TensorFlow in the AI‑For‑Beginners Curriculum

> Learn to implement object detection using TensorFlow with the AI-For-Beginners curriculum. Master RetinaNet, ResNet-50, Focal Loss, and NMS in Lesson 4.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-09

---

**The AI‑For‑Beginners curriculum teaches object detection using TensorFlow through a RetinaNet implementation with ResNet‑50 backbone, Focal Loss optimization, and Non‑Maximum Suppression post‑processing in Lesson 4 (Computer Vision).**

The microsoft/AI‑For‑Beginners repository provides a comprehensive pathway to implement object detection using TensorFlow within its dedicated Computer Vision track. Located in `lessons/4-ComputerVision/11-ObjectDetection/`, the lesson combines theoretical foundations—such as **Intersection‑over‑Union (IoU)** and **mean‑average‑precision (mAP)** metrics—with a practical implementation roadmap. Learners build an end‑to‑end pipeline using TensorFlow 2’s `tf.keras` API and `tf.data` utilities.

## Understanding the Curriculum Structure

The object detection lesson centers on two primary resources. The [`README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/README.md) in `lessons/4-ComputerVision/11-ObjectDetection/` explains evaluation metrics and algorithmic concepts, while the `ObjectDetection.ipynb` notebook contains executable code demonstrating TensorFlow imports and model construction. The curriculum also references [`etc/quiz-src/questions-en.txt`](https://github.com/microsoft/AI-For-Beginners/blob/main/etc/quiz-src/questions-en.txt) to reinforce understanding of bounding box formats and IoU calculations.

According to the lesson documentation, the recommended approach utilizes the **RetinaNet** architecture. The curriculum explicitly links to the official Keras RetinaNet tutorial (`https://keras.io/examples/vision/retinanet/`) and the optional `keras-retinanet` library (`https://github.com/fizyr/keras-retinanet`) for advanced implementations.

## RetinaNet Architecture Components

The TensorFlow implementation follows a six‑stage architecture:

1. **Backbone Network**: A **ResNet‑50** pretrained on ImageNet extracts multi‑scale features.
2. **Feature Pyramid Network (FPN)**: Processes backbone outputs to create feature maps at multiple resolutions.
3. **Detection Heads**: Parallel subnetworks for classification (object category probabilities) and regression (bounding box adjustments).
4. **Anchor Generation**: Predefined reference boxes at multiple scales and aspect ratios.
5. **Loss Computation**: **Focal Loss** reduces easy‑negative dominance in classification, while smooth‑L1 loss optimizes box regression.
6. **Post‑Processing**: **Non‑Maximum Suppression (NMS)** filters overlapping predictions using `tf.image` utilities.

## Implementing the TensorFlow Pipeline

### Data Preparation with tf.data

The curriculum demonstrates loading detection datasets like Hollywood Heads, stored with COCO‑style JSON annotations. Images are paired with bounding boxes in `[x_min, y_min, x_max, y_max]` format alongside class labels.

The implementation constructs a `tf.data` pipeline for efficient preprocessing:

```python
import tensorflow as tf
from tensorflow import keras
import pathlib
import json

def load_coco_annotations(json_path):
    with open(json_path) as f:
        data = json.load(f)
    images = {img['id']: img['file_name'] for img in data['images']}
    annotations = {}
    for ann in data['annotations']:
        img_id = ann['image_id']
        bbox = ann['bbox']  # [x, y, width, height]

        category = ann['category_id']
        annotations.setdefault(img_id, []).append((bbox, category))
    return images, annotations

def parse_example(img_path, ann):
    img = tf.io.read_file(img_path)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.resize(img, [640, 640])
    img = img / 255.0
    boxes = tf.convert_to_tensor([a[0] for a in ann], dtype=tf.float32)
    # Convert to normalized ymin, xmin, ymax, xmax

    boxes = tf.stack([
        boxes[:,1]/640, boxes[:,0]/640,
        (boxes[:,1]+boxes[:,3])/640, (boxes[:,0]+boxes[:,2])/640
    ], axis=-1)
    classes = tf.convert_to_tensor([a[1] for a in ann], dtype=tf.int32)
    return img, {"boxes": boxes, "classes": classes}

```

### Model Construction

Using the `keras-retinanet` library, the model instantiation combines the backbone and detection heads:

```python
from keras_retinanet import models
from keras_retinanet import losses

# Build RetinaNet with ResNet50 backbone

model = models.backbone('resnet50').retinanet(num_classes=1)

model.compile(
    optimizer=keras.optimizers.Adam(lr=1e-5),
    loss={
        'regression': losses.smooth_l1(),
        'classification': losses.focal()
    }
)

```

### Training Configuration

The training loop utilizes `tf.keras.Model.fit` with frozen or fine‑tuned backbone weights:

```python
def build_dataset(img_dir, annotations):
    img_paths = [str(p) for p in pathlib.Path(img_dir).glob("*.jpg")]
    dataset = tf.data.Dataset.from_tensor_slices((
        img_paths, 
        [annotations[p.name] for p in pathlib.Path(img_dir).glob("*.jpg")]
    ))
    dataset = dataset.map(parse_example, num_parallel_calls=tf.data.AUTOTUNE)
    return dataset.shuffle(1024).batch(8).prefetch(tf.data.AUTOTUNE)

train_ds = build_dataset("data/hollywood_heads/train", train_ann)
val_ds = build_dataset("data/hollywood_heads/val", val_ann)

model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(patience=3)]
)

```

### Post‑Processing and Inference

After `model.predict`, results are filtered by confidence threshold and visualized:

```python
import matplotlib.pyplot as plt

def visualize_predictions(image, preds, threshold=0.5):
    plt.figure()
    plt.imshow(image)
    for box, score in zip(preds['boxes'][0], preds['scores'][0]):
        if score < threshold: 
            continue
        y1, x1, y2, x2 = box
        plt.gca().add_patch(plt.Rectangle(
            (x1*image.shape[1], y1*image.shape[0]), 
            (x2-x1)*image.shape[1], (y2-y1)*image.shape[0],
            linewidth=2, edgecolor='red', facecolor='none'
        ))
    plt.show()

sample_img, _ = next(iter(val_ds))
preds = model.predict(sample_img)
visualize_predictions(sample_img[0].numpy(), preds)

```

## Complete Code Implementation

Below is the consolidated pipeline assuming a directory structure `data/hollywood_heads/` with `train/` and `val/` subfolders containing images and corresponding COCO‑style JSON annotation files:

```python

# 1️⃣ Imports

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import pathlib
import json

# 2️⃣ Load annotations

def load_coco_annotations(json_path):
    with open(json_path) as f:
        data = json.load(f)
    images = {img['id']: img['file_name'] for img in data['images']}
    annotations = {}
    for ann in data['annotations']:
        img_id = ann['image_id']
        bbox = ann['bbox']          # [x, y, width, height]

        category = ann['category_id']
        annotations.setdefault(img_id, []).append((bbox, category))
    return images, annotations

# 3️⃣ Dataset pipeline

def parse_example(img_path, ann):
    img = tf.io.read_file(img_path)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.resize(img, [640, 640])
    img = img / 255.0
    boxes = tf.convert_to_tensor([a[0] for a in ann], dtype=tf.float32)
    # Convert to normalized ymin, xmin, ymax, xmax

    boxes = tf.stack([boxes[:,1]/640,
                      boxes[:,0]/640,
                      (boxes[:,1]+boxes[:,3])/640,
                      (boxes[:,0]+boxes[:,2])/640], axis=-1)
    classes = tf.convert_to_tensor([a[1] for a in ann], dtype=tf.int32)
    return img, {"boxes": boxes, "classes": classes}

def build_dataset(img_dir, annotations):
    img_paths = [str(p) for p in pathlib.Path(img_dir).glob("*.jpg")]
    dataset = tf.data.Dataset.from_tensor_slices((img_paths, [annotations[p.name] for p in pathlib.Path(img_dir).glob("*.jpg")]))
    dataset = dataset.map(parse_example, num_parallel_calls=tf.data.AUTOTUNE)
    return dataset.shuffle(1024).batch(8).prefetch(tf.data.AUTOTUNE)

# 4️⃣ Model – RetinaNet (using keras‑retinanet)

# Install: pip install keras-retinanet

from keras_retinanet import models
from keras_retinanet import losses
from keras_retinanet import layers as retinanet_layers

# Build a RetinaNet with ResNet50 backbone

model = models.backbone('resnet50').retinanet(num_classes=1)

# Compile with focal loss + smooth L1 loss

model.compile(
    optimizer=keras.optimizers.Adam(lr=1e-5),
    loss={
        'regression': losses.smooth_l1(),
        'classification': losses.focal()
    }
)

# 5️⃣ Train

train_ds = build_dataset("data/hollywood_heads/train", train_ann)
val_ds   = build_dataset("data/hollywood_heads/val",   val_ann)

model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(patience=3)]
)

# 6️⃣ Inference & visualization

def visualize_predictions(image, preds, threshold=0.5):
    import matplotlib.pyplot as plt
    plt.figure()
    plt.imshow(image)
    for box, score in zip(preds['boxes'][0], preds['scores'][0]):
        if score < threshold: continue
        y1, x1, y2, x2 = box
        plt.gca().add_patch(plt.Rectangle(
            (x1*image.shape[1], y1*image.shape[0]), 
            (x2-x1)*image.shape[1], (y2-y1)*image.shape[0],
            linewidth=2, edgecolor='red', facecolor='none'))
    plt.show()

sample_img, _ = next(iter(val_ds))
preds = model.predict(sample_img)
visualize_predictions(sample_img[0].numpy(), preds)

```

## Summary

- **Curriculum Location**: Object detection resources reside in `lessons/4-ComputerVision/11-ObjectDetection/`, including the conceptual [`README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/README.md) and executable `ObjectDetection.ipynb`.
- **Architecture**: The implementation leverages **RetinaNet** with a ResNet‑50 backbone and **Feature Pyramid Network** for multi‑scale detection.
- **TensorFlow Advantages**: `tf.data` pipelines handle efficient preprocessing, while `tf.keras` enables custom loss functions like **Focal Loss** and built‑in NMS utilities.
- **Evaluation**: Model performance is measured using **IoU** for bounding box accuracy and **mAP** for overall detection quality.
- **Extensibility**: The pipeline supports dataset substitution (COCO, Pascal VOC) and backbone modification through the `keras-retinanet` library.

## Frequently Asked Questions

### What dataset format does the AI‑For‑Beginners curriculum recommend for object detection?

The curriculum demonstrates **COCO‑style JSON annotations**, where each entry contains bounding boxes formatted as `[x, y, width, height]` alongside category IDs. Images are typically organized in directory structures like `data/hollywood_heads/train/` and paired with their corresponding annotation files. The preprocessing pipeline converts these boxes to normalized `[ymin, xmin, ymax, xmax]` tensors required by RetinaNet.

### Why does the curriculum use RetinaNet instead of other object detection models?

RetinaNet provides an optimal balance between accuracy and computational efficiency for educational purposes. Its **Focal Loss** mechanism specifically addresses foreground‑background class imbalance inherent in object detection, while the **Feature Pyramid Network** architecture naturally handles objects at multiple scales. These characteristics make it ideal for learning foundational concepts before exploring more complex architectures like YOLO or DETR.

### How do I evaluate model performance using the curriculum's recommended metrics?

You calculate **Intersection‑over‑Union (IoU)** by dividing the overlap area between predicted and ground‑truth boxes by their union area. **Mean‑average‑precision (mAP)** aggregates precision‑recall curves across all classes. The `ObjectDetection.ipynb` notebook and [`etc/quiz-src/questions-en.txt`](https://github.com/microsoft/AI-For-Beginners/blob/main/etc/quiz-src/questions-en.txt) file provide practical examples of computing these metrics using TensorFlow operations on validation datasets.

### Can I fine‑tune the model on a custom dataset instead of Hollywood Heads?

Yes. Replace the dataset paths in `build_dataset()` with your custom image directories and generate COCO‑style JSON annotations for your specific classes. Update `num_classes` in the `retinanet()` constructor to match your category count. The `tf.data` pipeline remains identical regardless of dataset, allowing seamless substitution of COCO, Pascal VOC, or custom agricultural/medical imaging datasets.