How to Perform Object Detection with TensorFlow in the AI-For-Beginners Curriculum

The Microsoft AI-For-Beginners curriculum teaches TensorFlow-based object detection through a three-stage progression: naïve sliding-window classification with VGG-16, synthetic bounding-box regression, and references to production RetinaNet implementations.

The microsoft/AI-For-Beginners repository provides a structured introduction to computer vision using TensorFlow and Keras. In lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb, learners progress from basic classification techniques to building regression models that predict bounding-box coordinates. This hands-on approach demystifies how modern object detectors work by deconstructing the problem into manageable components.

Stage 1: Sliding-Window Detection with VGG-16

The curriculum begins with a naïve sliding-window approach that repurposes a pre-trained classifier for localization. Instead of training a new model, learners use keras.applications.vgg16.VGG16 (loaded with ImageNet weights) to evaluate overlapping tiles of an input image.

In the predict function implementation, the model preprocesses a 224×224 tile and returns the summed probability across ImageNet class indices 281–294 (the "cat" category). The predict_map function systematically slides this window across the image to build a heat-map of object presence.

from tensorflow import keras
import cv2, numpy as np, matplotlib.pyplot as plt

vgg = keras.applications.vgg16.VGG16(weights="imagenet")

def predict(img):
    im = cv2.resize(img, (224, 224))
    im = keras.applications.vgg16.preprocess_input(im)
    pr = vgg.predict(np.expand_dims(im, axis=0))[0]
    # ImageNet cat classes are 281-294

    return np.sum(pr[281:295])

def predict_map(img, n):
    dx = img.shape[0] // n
    heat = np.zeros((n, n), dtype=np.float32)
    for i in range(n):
        for j in range(n):
            tile = img[dx*i:dx*(i+1), dx*j:dx*(j+1)]
            heat[i, j] = predict(tile)
    return heat

This technique demonstrates how classification models can approximate localization through exhaustive search, though it remains computationally expensive for real-world applications.

Stage 2: Bounding-Box Regression on Synthetic Data

The second stage introduces the core concept of direct coordinate prediction using a custom regression network. The notebook generates synthetic 8×8 grayscale images containing black rectangles via the generate_images function, creating a controlled dataset for training.

A small fully-connected network built with keras.Sequential learns to predict the four bounding-box parameters [x, y, w, h]. The architecture flattens the input and passes it through a 200-unit ReLU layer with dropout before outputting normalized coordinates. The curriculum evaluates performance using a custom Intersection-over-Union (IoU) metric.

import numpy as np

def generate_images(num_imgs, img_size=8, min_sz=1, max_sz=4):
    bboxes = np.zeros((num_imgs, 4))
    imgs = np.zeros((num_imgs, img_size, img_size))
    for i in range(num_imgs):
        w, h = np.random.randint(min_sz, max_sz, size=2)
        x = np.random.randint(0, img_size - w)
        y = np.random.randint(0, img_size - h)
        imgs[i, x:x+w, y:y+h] = 1.  # draw rectangle

        bboxes[i] = [x, y, w, h]
    return imgs, bboxes

def IOU(b1, b2):
    x1, y1, w1, h1 = b1
    x2, y2, w2, h2 = b2
    w_i = min(x1 + w1, x2 + w2) - max(x1, x2)
    h_i = min(y1 + h1, y2 + h2) - max(y1, y2)
    if w_i <= 0 or h_i <= 0: 
        return 0.0
    inter = w_i * h_i
    union = w1 * h1 + w2 * h2 - inter
    return inter / union

# Generate and normalize data

imgs, bboxes = generate_images(100_000)
bb_norm = bboxes / 8.0  # scale to [0, 1]

# Build regression model

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(8, 8)),
    keras.layers.Dense(200, activation="relu"),
    keras.layers.Dropout(0.2),
    keras.layers.Dense(4)  # predicts [x, y, w, h]

])
model.compile(optimizer="sgd", loss="mse")
model.fit((imgs - np.mean(imgs)) / np.std(imgs), bb_norm, epochs=30)

This stage establishes the fundamental mechanics of modern single-shot detectors without the complexity of working with large-scale image datasets.

Stage 3: Production-Grade Detection with RetinaNet

After mastering the basics, the curriculum directs learners to production-ready architectures by referencing the official Keras RetinaNet implementation. Unlike the simplified regression model, RetinaNet incorporates a Feature Pyramid Network (FPN), focal loss, and multi-scale anchor boxes to handle objects at various scales and aspect ratios.

The notebook explicitly points to the Keras RetinaNet tutorial (keras.io/examples/vision/retinanet/) and the fizyr/keras-retinanet library. This transition illustrates how the simple regression concepts from Stage 2 scale into state-of-the-art detectors that process high-resolution images with multiple objects per frame.

Summary

• The AI-For-Beginners curriculum in microsoft/AI-For-Beginners teaches object detection through three progressive TensorFlow implementations.
• Stage 1 uses keras.applications.vgg16.VGG16 with a sliding-window strategy to build class activation heat-maps without training new weights.
• Stage 2 implements a custom keras.Sequential regressor on synthetic 8×8 images, predicting [x, y, w, h] coordinates with mean-squared error loss and evaluating with the IOU metric.
• Stage 3 bridges to production systems by referencing RetinaNet, demonstrating how feature pyramids and focal loss solve real-world detection challenges.
• All implementations are contained in lessons/4-ComputerVision/11-ObjectDetection/ObjectDetection.ipynb.

Frequently Asked Questions

What TensorFlow version does the AI-For-Beginners curriculum require?

The code examples in ObjectDetection.ipynb use the modern Keras API integrated into TensorFlow 2.x. The keras.applications module and Sequential API patterns shown are compatible with TensorFlow 2.4 and later versions.

Do I need a GPU to run the object detection examples?

The sliding-window VGG-16 demonstration and synthetic 8×8 regression model run efficiently on CPU. However, the RetinaNet production models referenced in Stage 3 require GPU acceleration for reasonable training times on real datasets.

How does the sliding-window approach differ from RetinaNet?

The sliding-window method in Stage 1 evaluates multiple image crops independently using a classifier, resulting in redundant computations. RetinaNet (Stage 3) uses a single-shot detector architecture that predicts bounding boxes and class probabilities in one forward pass through a feature pyramid network.

Where can I find the dataset used in the bounding-box regression example?

The curriculum generates synthetic data on-the-fly using the generate_images function, which creates random rectangles on 8×8 grids. This eliminates external dependencies and allows learners to focus purely on the regression mechanics without data preprocessing overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}