How to Implement Object Detection Using TensorFlow in the AI‑For‑Beginners Curriculum

The AI‑For‑Beginners curriculum teaches object detection using TensorFlow through a RetinaNet implementation with ResNet‑50 backbone, Focal Loss optimization, and Non‑Maximum Suppression post‑processing in Lesson 4 (Computer Vision).

The microsoft/AI‑For‑Beginners repository provides a comprehensive pathway to implement object detection using TensorFlow within its dedicated Computer Vision track. Located in lessons/4-ComputerVision/11-ObjectDetection/, the lesson combines theoretical foundations—such as Intersection‑over‑Union (IoU) and mean‑average‑precision (mAP) metrics—with a practical implementation roadmap. Learners build an end‑to‑end pipeline using TensorFlow 2’s tf.keras API and tf.data utilities.

Understanding the Curriculum Structure

The object detection lesson centers on two primary resources. The README.md in lessons/4-ComputerVision/11-ObjectDetection/ explains evaluation metrics and algorithmic concepts, while the ObjectDetection.ipynb notebook contains executable code demonstrating TensorFlow imports and model construction. The curriculum also references etc/quiz-src/questions-en.txt to reinforce understanding of bounding box formats and IoU calculations.

According to the lesson documentation, the recommended approach utilizes the RetinaNet architecture. The curriculum explicitly links to the official Keras RetinaNet tutorial (https://keras.io/examples/vision/retinanet/) and the optional keras-retinanet library (https://github.com/fizyr/keras-retinanet) for advanced implementations.

RetinaNet Architecture Components

The TensorFlow implementation follows a six‑stage architecture:

1. Backbone Network: A ResNet‑50 pretrained on ImageNet extracts multi‑scale features.
2. Feature Pyramid Network (FPN): Processes backbone outputs to create feature maps at multiple resolutions.
3. Detection Heads: Parallel subnetworks for classification (object category probabilities) and regression (bounding box adjustments).
4. Anchor Generation: Predefined reference boxes at multiple scales and aspect ratios.
5. Loss Computation: Focal Loss reduces easy‑negative dominance in classification, while smooth‑L1 loss optimizes box regression.
6. Post‑Processing: Non‑Maximum Suppression (NMS) filters overlapping predictions using tf.image utilities.

Implementing the TensorFlow Pipeline

Data Preparation with tf.data

The curriculum demonstrates loading detection datasets like Hollywood Heads, stored with COCO‑style JSON annotations. Images are paired with bounding boxes in [x_min, y_min, x_max, y_max] format alongside class labels.

The implementation constructs a tf.data pipeline for efficient preprocessing:

import tensorflow as tf
from tensorflow import keras
import pathlib
import json

def load_coco_annotations(json_path):
    with open(json_path) as f:
        data = json.load(f)
    images = {img['id']: img['file_name'] for img in data['images']}
    annotations = {}
    for ann in data['annotations']:
        img_id = ann['image_id']
        bbox = ann['bbox']  # [x, y, width, height]

        category = ann['category_id']
        annotations.setdefault(img_id, []).append((bbox, category))
    return images, annotations

def parse_example(img_path, ann):
    img = tf.io.read_file(img_path)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.resize(img, [640, 640])
    img = img / 255.0
    boxes = tf.convert_to_tensor([a[0] for a in ann], dtype=tf.float32)
    # Convert to normalized ymin, xmin, ymax, xmax

    boxes = tf.stack([
        boxes[:,1]/640, boxes[:,0]/640,
        (boxes[:,1]+boxes[:,3])/640, (boxes[:,0]+boxes[:,2])/640
    ], axis=-1)
    classes = tf.convert_to_tensor([a[1] for a in ann], dtype=tf.int32)
    return img, {"boxes": boxes, "classes": classes}

Model Construction

Using the keras-retinanet library, the model instantiation combines the backbone and detection heads:

from keras_retinanet import models
from keras_retinanet import losses

# Build RetinaNet with ResNet50 backbone

model = models.backbone('resnet50').retinanet(num_classes=1)

model.compile(
    optimizer=keras.optimizers.Adam(lr=1e-5),
    loss={
        'regression': losses.smooth_l1(),
        'classification': losses.focal()
    }
)

Training Configuration

The training loop utilizes tf.keras.Model.fit with frozen or fine‑tuned backbone weights:

def build_dataset(img_dir, annotations):
    img_paths = [str(p) for p in pathlib.Path(img_dir).glob("*.jpg")]
    dataset = tf.data.Dataset.from_tensor_slices((
        img_paths, 
        [annotations[p.name] for p in pathlib.Path(img_dir).glob("*.jpg")]
    ))
    dataset = dataset.map(parse_example, num_parallel_calls=tf.data.AUTOTUNE)
    return dataset.shuffle(1024).batch(8).prefetch(tf.data.AUTOTUNE)

train_ds = build_dataset("data/hollywood_heads/train", train_ann)
val_ds = build_dataset("data/hollywood_heads/val", val_ann)

model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(patience=3)]
)

Post‑Processing and Inference

After model.predict, results are filtered by confidence threshold and visualized:

import matplotlib.pyplot as plt

def visualize_predictions(image, preds, threshold=0.5):
    plt.figure()
    plt.imshow(image)
    for box, score in zip(preds['boxes'][0], preds['scores'][0]):
        if score < threshold: 
            continue
        y1, x1, y2, x2 = box
        plt.gca().add_patch(plt.Rectangle(
            (x1*image.shape[1], y1*image.shape[0]), 
            (x2-x1)*image.shape[1], (y2-y1)*image.shape[0],
            linewidth=2, edgecolor='red', facecolor='none'
        ))
    plt.show()

sample_img, _ = next(iter(val_ds))
preds = model.predict(sample_img)
visualize_predictions(sample_img[0].numpy(), preds)

Complete Code Implementation

Below is the consolidated pipeline assuming a directory structure data/hollywood_heads/ with train/ and val/ subfolders containing images and corresponding COCO‑style JSON annotation files:

# 1️⃣ Imports

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import pathlib
import json

# 2️⃣ Load annotations

def load_coco_annotations(json_path):
    with open(json_path) as f:
        data = json.load(f)
    images = {img['id']: img['file_name'] for img in data['images']}
    annotations = {}
    for ann in data['annotations']:
        img_id = ann['image_id']
        bbox = ann['bbox']          # [x, y, width, height]

        category = ann['category_id']
        annotations.setdefault(img_id, []).append((bbox, category))
    return images, annotations

# 3️⃣ Dataset pipeline

def parse_example(img_path, ann):
    img = tf.io.read_file(img_path)
    img = tf.image.decode_jpeg(img, channels=3)
    img = tf.image.resize(img, [640, 640])
    img = img / 255.0
    boxes = tf.convert_to_tensor([a[0] for a in ann], dtype=tf.float32)
    # Convert to normalized ymin, xmin, ymax, xmax

    boxes = tf.stack([boxes[:,1]/640,
                      boxes[:,0]/640,
                      (boxes[:,1]+boxes[:,3])/640,
                      (boxes[:,0]+boxes[:,2])/640], axis=-1)
    classes = tf.convert_to_tensor([a[1] for a in ann], dtype=tf.int32)
    return img, {"boxes": boxes, "classes": classes}

def build_dataset(img_dir, annotations):
    img_paths = [str(p) for p in pathlib.Path(img_dir).glob("*.jpg")]
    dataset = tf.data.Dataset.from_tensor_slices((img_paths, [annotations[p.name] for p in pathlib.Path(img_dir).glob("*.jpg")]))
    dataset = dataset.map(parse_example, num_parallel_calls=tf.data.AUTOTUNE)
    return dataset.shuffle(1024).batch(8).prefetch(tf.data.AUTOTUNE)

# 4️⃣ Model – RetinaNet (using keras‑retinanet)

# Install: pip install keras-retinanet

from keras_retinanet import models
from keras_retinanet import losses
from keras_retinanet import layers as retinanet_layers

# Build a RetinaNet with ResNet50 backbone

model = models.backbone('resnet50').retinanet(num_classes=1)

# Compile with focal loss + smooth L1 loss

model.compile(
    optimizer=keras.optimizers.Adam(lr=1e-5),
    loss={
        'regression': losses.smooth_l1(),
        'classification': losses.focal()
    }
)

# 5️⃣ Train

train_ds = build_dataset("data/hollywood_heads/train", train_ann)
val_ds   = build_dataset("data/hollywood_heads/val",   val_ann)

model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=20,
    callbacks=[keras.callbacks.EarlyStopping(patience=3)]
)

# 6️⃣ Inference & visualization

def visualize_predictions(image, preds, threshold=0.5):
    import matplotlib.pyplot as plt
    plt.figure()
    plt.imshow(image)
    for box, score in zip(preds['boxes'][0], preds['scores'][0]):
        if score < threshold: continue
        y1, x1, y2, x2 = box
        plt.gca().add_patch(plt.Rectangle(
            (x1*image.shape[1], y1*image.shape[0]), 
            (x2-x1)*image.shape[1], (y2-y1)*image.shape[0],
            linewidth=2, edgecolor='red', facecolor='none'))
    plt.show()

sample_img, _ = next(iter(val_ds))
preds = model.predict(sample_img)
visualize_predictions(sample_img[0].numpy(), preds)

Summary

• Curriculum Location: Object detection resources reside in lessons/4-ComputerVision/11-ObjectDetection/, including the conceptual README.md and executable ObjectDetection.ipynb.
• Architecture: The implementation leverages RetinaNet with a ResNet‑50 backbone and Feature Pyramid Network for multi‑scale detection.
• TensorFlow Advantages: tf.data pipelines handle efficient preprocessing, while tf.keras enables custom loss functions like Focal Loss and built‑in NMS utilities.
• Evaluation: Model performance is measured using IoU for bounding box accuracy and mAP for overall detection quality.
• Extensibility: The pipeline supports dataset substitution (COCO, Pascal VOC) and backbone modification through the keras-retinanet library.

Frequently Asked Questions

What dataset format does the AI‑For‑Beginners curriculum recommend for object detection?

The curriculum demonstrates COCO‑style JSON annotations, where each entry contains bounding boxes formatted as [x, y, width, height] alongside category IDs. Images are typically organized in directory structures like data/hollywood_heads/train/ and paired with their corresponding annotation files. The preprocessing pipeline converts these boxes to normalized [ymin, xmin, ymax, xmax] tensors required by RetinaNet.

Why does the curriculum use RetinaNet instead of other object detection models?

RetinaNet provides an optimal balance between accuracy and computational efficiency for educational purposes. Its Focal Loss mechanism specifically addresses foreground‑background class imbalance inherent in object detection, while the Feature Pyramid Network architecture naturally handles objects at multiple scales. These characteristics make it ideal for learning foundational concepts before exploring more complex architectures like YOLO or DETR.

How do I evaluate model performance using the curriculum's recommended metrics?

You calculate Intersection‑over‑Union (IoU) by dividing the overlap area between predicted and ground‑truth boxes by their union area. Mean‑average‑precision (mAP) aggregates precision‑recall curves across all classes. The ObjectDetection.ipynb notebook and etc/quiz-src/questions-en.txt file provide practical examples of computing these metrics using TensorFlow operations on validation datasets.

Can I fine‑tune the model on a custom dataset instead of Hollywood Heads?

Yes. Replace the dataset paths in build_dataset() with your custom image directories and generate COCO‑style JSON annotations for your specific classes. Update num_classes in the retinanet() constructor to match your category count. The tf.data pipeline remains identical regardless of dataset, allowing seamless substitution of COCO, Pascal VOC, or custom agricultural/medical imaging datasets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}