# How to Implement Transfer Learning with ResNet50 and VGG16 in AI for Beginners Computer Vision

> Master transfer learning for computer vision using ResNet50 and VGG16. Reuse pretrained models, freeze layers, and train a new head for high accuracy on custom datasets with minimal compute.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Transfer learning lets you reuse pretrained ResNet50 or VGG16 convolutional layers from ImageNet, freeze them, and train only a new classification head to achieve high accuracy on small custom datasets with minimal compute.**

This guide walks through the exact implementation patterns used in the Microsoft AI for Beginners curriculum. You will learn how to load pretrained weights, freeze feature extractors, replace top classifiers, and fine-tune models using both PyTorch and TensorFlow—the two frameworks covered in `lessons/4-ComputerVision/08-TransferLearning/`.

## Why ResNet50 and VGG16 for Transfer Learning

The AI for Beginners course selects **VGG16/VGG19** and **ResNet50/ResNet101** as the primary architectures for transfer learning demonstrations. Each serves a distinct pedagogical purpose:

- **VGG16**: Simple, sequential architecture that makes it easy to understand layer-by-layer feature extraction. Ideal for first-time learners.
- **ResNet50**: Deeper architecture with skip connections that achieves higher accuracy at the cost of increased computation. Better for production scenarios.

According to the curriculum's [`README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/README.md) in `lessons/4-ComputerVision/08-TransferLearning/`, both models are pretrained on ImageNet and exposed through standard library APIs in PyTorch (`torchvision.models`) and TensorFlow (`tf.keras.applications`).

## The Four-Step Transfer Learning Workflow

Every implementation in the course follows this pattern:

1. **Load the pretrained base** with ImageNet weights.
2. **Freeze convolutional layers** to preserve learned features.
3. **Replace the classifier head** to match your class count.
4. **Train only the new layers** (optionally unfreeze later for fine-tuning).

This approach trains in minutes on CPU while leveraging representations learned from millions of images.

## PyTorch Implementation with ResNet50

The notebook `TransferLearningPyTorch.ipynb` demonstrates ResNet50 transfer learning for binary classification (cats vs. dogs).

### Loading and Freezing the Base

```python
import torch
import torch.nn as nn
import torchvision.models as models

# Load pretrained ResNet50 from torchvision

model = models.resnet50(weights='IMAGENET1K_V2')

# Freeze all convolutional parameters

for param in model.parameters():
    param.requires_grad = False

```

### Replacing the Classifier Head

ResNet50's original `fc` layer outputs 1000 ImageNet classes. Replace it with a linear layer for your task:

```python
num_classes = 2  # cats vs. dogs

model.fc = nn.Linear(model.fc.in_features, num_classes)

```

### Complete Training Setup

```python
from torchvision import transforms, datasets
from torch.utils.data import DataLoader

# Image preprocessing matching ResNet50 training

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

# Load dataset from folder structure

train_ds = datasets.ImageFolder('data/train', transform=transform)
train_loader = DataLoader(train_ds, batch_size=32, shuffle=True)

# Optimize only the new classifier head

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)

# Training loop

model.train()
for epoch in range(5):
    for images, labels in train_loader:
        outputs = model(images)
        loss = criterion(outputs, labels)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print(f'Epoch {epoch+1} complete')

```

The key optimization is `model.fc.parameters()`—only the new layer receives gradients, making training fast even with limited data.

## TensorFlow/Keras Implementation with VGG16

The companion notebook `TransferLearningTF.ipynb` uses `tf.keras.applications` for the same workflow.

### Loading VGG16 Without the Top

```python
import tensorflow as tf
from tensorflow.keras.applications import VGG16
from tensorflow.keras import layers, models

# Load convolutional base only, frozen

base_model = VGG16(
    weights='imagenet',
    include_top=False,  # exclude original 1000-way classifier

    input_shape=(224, 224, 3)
)
base_model.trainable = False  # freeze entire base

```

### Building the Custom Head

```python
inputs = tf.keras.Input(shape=(224, 224, 3))

# Run frozen base in inference mode (affects BatchNorm)

x = base_model(inputs, training=False)

# Global pooling reduces spatial dimensions efficiently

x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(256, activation='relu')(x)
outputs = layers.Dense(2, activation='softmax')(x)

model = models.Model(inputs, outputs)

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

```

### Data Loading and Training

```python
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Rescale and split validation data

train_gen = ImageDataGenerator(
    rescale=1./255,
    validation_split=0.2
)

train_ds = train_gen.flow_from_directory(
    'data',
    target_size=(224, 224),
    batch_size=32,
    class_mode='sparse',
    subset='training'
)

val_ds = train_gen.flow_from_directory(
    'data',
    target_size=(224, 224),
    batch_size=32,
    class_mode='sparse',
    subset='validation'
)

# Train only the new head

model.fit(train_ds, epochs=5, validation_data=val_ds)

```

The `training=False` flag in `base_model(inputs, training=False)` is critical—it ensures BatchNorm layers use pretrained moving averages rather than batch statistics.

## Optional Fine-Tuning: Unfreezing Layers

After the new head converges, you can unfreeze deeper layers for additional accuracy. The curriculum's [`TrainingTricks.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/TrainingTricks.md) covers this advanced pattern:

```python

# Unfreeze the entire model for fine-tuning

base_model.trainable = True

# Or selectively unfreeze: last 30 layers of ResNet50

for param in model[:-30].parameters():
    param.requires_grad = False

# Use a much lower learning rate to preserve pretrained features

optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

```

## Key File Reference

| Path | Purpose |
|------|---------|
| [`lessons/4-ComputerVision/08-TransferLearning/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/README.md) | Conceptual overview and dataset instructions |
| `lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb` | Full ResNet50 PyTorch implementation |
| `lessons/4-ComputerVision/08-TransferLearning/TransferLearningTF.ipynb` | Full VGG16/ResNet50 TensorFlow implementation |
| `lessons/4-ComputerVision/08-TransferLearning/lab/OxfordPets.ipynb` | Extended lab with Oxford-IIIT Pets dataset |
| [`lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/08-TransferLearning/TrainingTricks.md) | Learning rate schedules and data augmentation tips |
| [`lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/4-ComputerVision/07-ConvNets/CNN_Architectures.md) | Architecture deep-dives for VGG and ResNet |

## Summary

- **Transfer learning with ResNet50 and VGG16** reuses ImageNet-trained convolutional layers to solve new image classification tasks with small datasets.
- **Freeze the base model** to preserve generic visual features (edges, textures, shapes) learned from millions of images.
- **Replace and train only the classifier head** for fast convergence—typically 5-10 epochs suffice.
- **PyTorch** uses `model.parameters()` control and `nn.Linear` replacement; **TensorFlow** uses `include_top=False` and functional API composition.
- Both frameworks expose pretrained models through `torchvision.models` and `tf.keras.applications` respectively.

## Frequently Asked Questions

### Should I use ResNet50 or VGG16 for my first transfer learning project?

Start with **VGG16**. Its straightforward sequential architecture makes debugging and understanding layer transformations easier. ResNet50's skip connections and greater depth add complexity that pays off only when you need maximum accuracy or are comfortable with the basics.

### Why must I use `training=False` when calling the frozen base in TensorFlow?

VGG16 includes **BatchNormalization** layers. When `training=True`, these layers recalculate mean and variance from your small dataset, destroying the pretrained statistics. `training=False` preserves the ImageNet-learned normalization parameters that stabilize early-layer features.

### How many images do I need for transfer learning to work effectively?

The AI for Beginners examples use **2,000 training images** (1,000 per class for cats vs. dogs). In practice, transfer learning yields usable results with **as few as 100-200 images per class**, though accuracy improves with more data. The pretrained base provides the heavy feature extraction; your data only needs to teach the final decision boundary.

### What input size should I use for ResNet50 and VGG16?

Both architectures expect **224×224 pixel RGB images** by default. VGG16 can accept other sizes when `include_top=False`; ResNet50's fully-connected layer requires fixed dimensions unless you modify the architecture. The curriculum standardizes on 224×224 for consistency with original ImageNet training.