How Transfer Learning Works in Computer Vision with Pre‑Trained Networks

Transfer learning leverages a neural network trained on a large dataset (such as ImageNet) and adapts it to a new, related task by freezing the convolutional backbone and replacing the classification head.

The AI‑For‑Beginners curriculum by Microsoft demonstrates this technique in lesson 08‑TransferLearning, where students build pet‑breed classifiers using TensorFlow/Keras and PyTorch. This approach dramatically reduces training time and improves accuracy when working with limited data.

---

Core Concepts of Transfer Learning in Computer Vision

Transfer learning exploits the hierarchical nature of visual features learned by deep convolutional networks. Early layers detect edges and textures; deeper layers recognize object parts and semantic concepts. Rather than training from scratch, you reuse these learned representations.

Feature Extraction vs. Fine‑Tuning

Two primary strategies appear in the curriculum:

• Feature extraction – freeze the entire pre‑trained backbone and use it as a fixed feature extractor. Only the new classification head trains on your data.
• Fine‑tuning – after initial training of the head, unfreeze some top layers and continue training at a reduced learning rate to adapt generic features to your domain.

Both strategies appear in the lesson's README at /lessons/4-ComputerVision/08‑TransferLearning/README.md.

---

Step‑by‑Step Transfer Learning Workflow

The AI‑For‑Beginners lesson structures transfer learning into four concrete steps, demonstrated on the 37‑class Oxford‑IIIT pet dataset.

1. Load a Pre‑Trained Base Model

Convolutional networks like MobileNetV2 (TensorFlow) or ResNet‑18 (PyTorch) come with weights trained on ImageNet's 1.2 million images. These weights encode rich visual priors.

2. Freeze the Base Layers

Freezing prevents backpropagation from modifying the pre‑trained weights. In PyTorch this means setting requires_grad = False on base parameters; in Keras you set base_model.trainable = False.

3. Replace the Classification Head

The original head (typically 1000 outputs for ImageNet) is replaced with a new dense layer matching your target classes:

# TensorFlow/Keras pattern

outputs = layers.Dense(NUM_CLASSES, activation="softmax")(x)

# PyTorch pattern

base_model.fc = nn.Linear(num_ftrs, NUM_CLASSES)

4. Fine‑Tune (Optional)

After the new head converges, selectively unfreeze layers and train with a small learning rate (e.g., 1e‑5) to refine features for your specific domain.

---

TensorFlow/Keras Implementation with MobileNetV2

The TransferLearningTF.ipynb notebook implements the complete pipeline. Below is the essential structure:

import tensorflow as tf
from tensorflow.keras import layers, models

# 1. Load pre‑trained MobileNetV2 without the top classification layer

base_model = tf.keras.applications.MobileNetV2(
    weights="imagenet", 
    include_top=False, 
    input_shape=(224, 224, 3)
)

# 2. Freeze the backbone

base_model.trainable = False

# 3. Build new classification head with dropout regularization

inputs = tf.keras.Input(shape=(224, 224, 3))
x = tf.keras.applications.mobilenet_v2.preprocess_input(inputs)
x = base_model(x, training=False)          # inference mode for frozen layers

x = layers.GlobalAveragePooling2D()(x)      # flatten spatial features

x = layers.Dropout(0.2)(x)                  # prevent overfitting

outputs = layers.Dense(NUM_CLASSES, activation="softmax")(x)

model = models.Model(inputs, outputs)

# 4. Compile with higher learning rate for the new head

model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

model.fit(train_dataset, epochs=5, validation_data=val_dataset)

The notebook uses GlobalAveragePooling2D to reduce spatial dimensions, avoiding the parameter explosion of a fully‑connected layer. Dropout (rate 0.2) provides regularization for the small pet dataset.

---

PyTorch Implementation with ResNet‑18

The PyTorch counterpart in TransferLearningPyTorch.ipynb follows the same principles with framework‑specific syntax:

import torch
import torchvision
from torch import nn, optim

# 1. Load pre‑trained ResNet‑18

base_model = torchvision.models.resnet18(weights="IMAGENET1K_V1")

# 2. Freeze all base parameters

for param in base_model.parameters():
    param.requires_grad = False

# 3. Replace the final fully‑connected layer

num_ftrs = base_model.fc.in_features
base_model.fc = nn.Sequential(
    nn.Dropout(0.2),
    nn.Linear(num_ftrs, NUM_CLASSES)  # 37 for Oxford pets

)

# 4. Optimize only the new head's parameters

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(base_model.fc.parameters(), lr=1e-3)

# 5. Standard training loop

base_model.train()
for epoch in range(5):
    for imgs, labels in train_loader:
        optimizer.zero_grad()
        outputs = base_model(imgs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

Note the explicit requires_grad = False loop freezing base parameters and the targeted optimizer that only receives base_model.fc.parameters().

---

Why Transfer Learning Works: The Source Code Evidence

The AI‑For‑Beginners materials explain the theoretical basis in /lessons/4-ComputerVision/08‑TransferLearning/README.md:

• Speed: Convergence occurs in minutes because convolutional filters are already well‑initialized from millions of images.
• Data efficiency: Small datasets (hundreds to thousands of images) achieve usable accuracy that would be impossible with random initialization.
• Computational savings: Only the new head's parameters update during initial training, reducing memory and compute requirements.

The lesson's TrainingTricks.md file (located at /lessons/4-ComputerVision/08‑TransferLearning/TrainingTricks.md) documents complementary techniques: data augmentation, learning‑rate schedules, and progressive unfreezing for fine‑tuning.

---

Practical Lab Exercise

The repository includes a hands‑on lab at /lessons/4-ComputerVision/08‑TransferLearning/lab/README.md where students:

1. Download the Oxford‑IIIT pet dataset (37 breed categories)
2. Implement both feature‑extraction and fine‑tuning pipelines
3. Compare validation accuracy between frozen and partially‑unfrozen models

This reinforces the curriculum's emphasis on empirical understanding of when to apply each transfer‑learning strategy.

---

Summary

• Transfer learning in computer vision reuses pre‑trained convolutional features from models like MobileNetV2 and ResNet‑18.
• Freezing base layers preserves generic visual knowledge while a new classification head learns task‑specific patterns.
• Fine‑tuning top layers with low learning rates adapts features to domain specifics without catastrophic forgetting.
• The AI‑For‑Beginners lesson 08‑TransferLearning provides complete, runnable implementations in both TensorFlow/Keras and PyTorch on the Oxford‑IIIT pet classification task.

---

Frequently Asked Questions

What is the difference between feature extraction and fine‑tuning in transfer learning?

Feature extraction freezes all pre‑trained layers and uses them as a fixed feature extractor, training only the new classification head. Fine‑tuning unfreezes some or all pre‑trained layers and continues training at a reduced learning rate to adapt the learned representations to the new dataset. According to the AI‑For‑Beginners curriculum, you should start with feature extraction and only proceed to fine‑tuning if you have sufficient data and observe underfitting.

Why use GlobalAveragePooling2D instead of Flatten in transfer learning?

GlobalAveragePooling2D reduces each spatial feature map to a single value by averaging, producing a compact vector representation. This approach has fewer parameters than Flatten followed by a dense layer, reducing overfitting risk on small datasets like the 37‑class Oxford pets. The TransferLearningTF.ipynb notebook implements this pattern specifically for this reason.

How do you know which layers to unfreeze when fine‑tuning?

Unfreeze the top (deepest) convolutional blocks first, as these contain task‑specific features, while keeping early layers frozen as they encode generic, low‑level patterns. The TrainingTricks.md file in the AI‑For‑Beginners repository recommends unfreezing progressively and using learning rates 10× smaller than for the initial head training to avoid destroying pre‑trained weights.

Can transfer learning work with very small datasets (fewer than 100 images)?

Yes, though you should employ stronger regularization: lower learning rates, aggressive data augmentation, and possibly keeping all base layers frozen. The Oxford‑IIIT pet dataset in the curriculum has approximately 200 images per class, which is already sufficient for strong performance with a frozen MobileNetV2 backbone. For smaller datasets, consider adding more dropout or using a simpler classification head.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}