How to Implement Transfer Learning with ResNet50 and VGG16 in AI for Beginners Computer Vision

Transfer learning lets you reuse pretrained ResNet50 or VGG16 convolutional layers from ImageNet, freeze them, and train only a new classification head to achieve high accuracy on small custom datasets with minimal compute.

This guide walks through the exact implementation patterns used in the Microsoft AI for Beginners curriculum. You will learn how to load pretrained weights, freeze feature extractors, replace top classifiers, and fine-tune models using both PyTorch and TensorFlow—the two frameworks covered in lessons/4-ComputerVision/08-TransferLearning/.

Why ResNet50 and VGG16 for Transfer Learning

The AI for Beginners course selects VGG16/VGG19 and ResNet50/ResNet101 as the primary architectures for transfer learning demonstrations. Each serves a distinct pedagogical purpose:

• VGG16: Simple, sequential architecture that makes it easy to understand layer-by-layer feature extraction. Ideal for first-time learners.
• ResNet50: Deeper architecture with skip connections that achieves higher accuracy at the cost of increased computation. Better for production scenarios.

According to the curriculum's README.md in lessons/4-ComputerVision/08-TransferLearning/, both models are pretrained on ImageNet and exposed through standard library APIs in PyTorch (torchvision.models) and TensorFlow (tf.keras.applications).

The Four-Step Transfer Learning Workflow

Every implementation in the course follows this pattern:

1. Load the pretrained base with ImageNet weights.
2. Freeze convolutional layers to preserve learned features.
3. Replace the classifier head to match your class count.
4. Train only the new layers (optionally unfreeze later for fine-tuning).

This approach trains in minutes on CPU while leveraging representations learned from millions of images.

PyTorch Implementation with ResNet50

The notebook TransferLearningPyTorch.ipynb demonstrates ResNet50 transfer learning for binary classification (cats vs. dogs).

Loading and Freezing the Base

import torch
import torch.nn as nn
import torchvision.models as models

# Load pretrained ResNet50 from torchvision

model = models.resnet50(weights='IMAGENET1K_V2')

# Freeze all convolutional parameters

for param in model.parameters():
    param.requires_grad = False

Replacing the Classifier Head

ResNet50's original fc layer outputs 1000 ImageNet classes. Replace it with a linear layer for your task:

num_classes = 2  # cats vs. dogs

model.fc = nn.Linear(model.fc.in_features, num_classes)

Complete Training Setup

from torchvision import transforms, datasets
from torch.utils.data import DataLoader

# Image preprocessing matching ResNet50 training

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

# Load dataset from folder structure

train_ds = datasets.ImageFolder('data/train', transform=transform)
train_loader = DataLoader(train_ds, batch_size=32, shuffle=True)

# Optimize only the new classifier head

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)

# Training loop

model.train()
for epoch in range(5):
    for images, labels in train_loader:
        outputs = model(images)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f'Epoch {epoch+1} complete')

The key optimization is model.fc.parameters()—only the new layer receives gradients, making training fast even with limited data.

TensorFlow/Keras Implementation with VGG16

The companion notebook TransferLearningTF.ipynb uses tf.keras.applications for the same workflow.

Loading VGG16 Without the Top

import tensorflow as tf
from tensorflow.keras.applications import VGG16
from tensorflow.keras import layers, models

# Load convolutional base only, frozen

base_model = VGG16(
    weights='imagenet',
    include_top=False,  # exclude original 1000-way classifier

    input_shape=(224, 224, 3)
)
base_model.trainable = False  # freeze entire base

Building the Custom Head

inputs = tf.keras.Input(shape=(224, 224, 3))

# Run frozen base in inference mode (affects BatchNorm)

x = base_model(inputs, training=False)

# Global pooling reduces spatial dimensions efficiently

x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(256, activation='relu')(x)
outputs = layers.Dense(2, activation='softmax')(x)

model = models.Model(inputs, outputs)

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

Data Loading and Training

from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Rescale and split validation data

train_gen = ImageDataGenerator(
    rescale=1./255,
    validation_split=0.2
)

train_ds = train_gen.flow_from_directory(
    'data',
    target_size=(224, 224),
    batch_size=32,
    class_mode='sparse',
    subset='training'
)

val_ds = train_gen.flow_from_directory(
    'data',
    target_size=(224, 224),
    batch_size=32,
    class_mode='sparse',
    subset='validation'
)

# Train only the new head

model.fit(train_ds, epochs=5, validation_data=val_ds)

The training=False flag in base_model(inputs, training=False) is critical—it ensures BatchNorm layers use pretrained moving averages rather than batch statistics.

Optional Fine-Tuning: Unfreezing Layers

After the new head converges, you can unfreeze deeper layers for additional accuracy. The curriculum's TrainingTricks.md covers this advanced pattern:

# Unfreeze the entire model for fine-tuning

base_model.trainable = True

# Or selectively unfreeze: last 30 layers of ResNet50

for param in model[:-30].parameters():
    param.requires_grad = False

# Use a much lower learning rate to preserve pretrained features

optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

Key File Reference

Path	Purpose
lessons/4-ComputerVision/08-TransferLearning/README.md	Conceptual overview and dataset instructions
lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb	Full ResNet50 PyTorch implementation
lessons/4-ComputerVision/08-TransferLearning/TransferLearningTF.ipynb	Full VGG16/ResNet50 TensorFlow implementation
lessons/4-ComputerVision/08-TransferLearning/lab/OxfordPets.ipynb	Extended lab with Oxford-IIIT Pets dataset
lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md	Learning rate schedules and data augmentation tips
lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md	Architecture deep-dives for VGG and ResNet

Summary

• Transfer learning with ResNet50 and VGG16 reuses ImageNet-trained convolutional layers to solve new image classification tasks with small datasets.
• Freeze the base model to preserve generic visual features (edges, textures, shapes) learned from millions of images.
• Replace and train only the classifier head for fast convergence—typically 5-10 epochs suffice.
• PyTorch uses model.parameters() control and nn.Linear replacement; TensorFlow uses include_top=False and functional API composition.
• Both frameworks expose pretrained models through torchvision.models and tf.keras.applications respectively.

Frequently Asked Questions

Should I use ResNet50 or VGG16 for my first transfer learning project?

Start with VGG16. Its straightforward sequential architecture makes debugging and understanding layer transformations easier. ResNet50's skip connections and greater depth add complexity that pays off only when you need maximum accuracy or are comfortable with the basics.

Why must I use training=False when calling the frozen base in TensorFlow?

VGG16 includes BatchNormalization layers. When training=True, these layers recalculate mean and variance from your small dataset, destroying the pretrained statistics. training=False preserves the ImageNet-learned normalization parameters that stabilize early-layer features.

How many images do I need for transfer learning to work effectively?

The AI for Beginners examples use 2,000 training images (1,000 per class for cats vs. dogs). In practice, transfer learning yields usable results with as few as 100-200 images per class, though accuracy improves with more data. The pretrained base provides the heavy feature extraction; your data only needs to teach the final decision boundary.

What input size should I use for ResNet50 and VGG16?

Both architectures expect 224×224 pixel RGB images by default. VGG16 can accept other sizes when include_top=False; ResNet50's fully-connected layer requires fixed dimensions unless you modify the architecture. The curriculum standardizes on 224×224 for consistency with original ImageNet training.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}