# How Transfer Learning Works When Fine-Tuning Frozen vs Unfrozen Layers

> Explore transfer learning strategies in ai-engineering-from-scratch. Learn how fine-tuning frozen vs unfrozen layers impacts GPU memory and model accuracy for optimal performance.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-12

---

**Transfer learning adapts pre-trained models by either freezing the base architecture to train only the classification head (minimizing GPU memory and preventing catastrophic forgetting) or unfreezing all layers for full fine-tuning (maximizing accuracy when downstream tasks diverge from pre-training data).**

The `rohitg00/ai-engineering-from-scratch` curriculum demonstrates practical transfer learning through a toggle mechanism that switches between frozen-body and full fine-tuning regimes. This approach allows practitioners to leverage massive pre-trained models while controlling computational costs and preserving general knowledge based on dataset size and domain similarity.

## Freezing the Body: Head-Only Fine-Tuning

Freezing the model body involves setting `requires_grad=False` on all parameters except the final classification head, effectively treating the pre-trained layers as a fixed feature extractor.

### Why Freeze the Base Layers?

The curriculum emphasizes that freezing preserves the **broad linguistic or visual features** learned during massive pre-training, which is critical when working with limited downstream data. As noted in the classifier fine-tuning documentation, "head-only training is fast, almost free in memory, and rarely overfits with this little data."

### Memory and Computational Efficiency

When the body is frozen, the optimizer stores gradients only for head parameters. This reduces GPU memory requirements by roughly an order of magnitude and accelerates each training step, making it feasible to fine-tune large models on consumer hardware.

## Unfrozen Full Fine-Tuning

Unfreezing the entire architecture allows gradients to flow through all layers, enabling the model to adapt its internal representations to the new domain.

### Task-Specific Adaptation

Full fine-tuning reaches higher accuracy when the downstream domain drifts significantly from the pre-training corpus. The curriculum observes that "full fine-tuning… reaches higher accuracy when the downstream domain drifts from the pretraining corpus," as the model can adjust low-level features to match new task distributions.

### Capacity for Large Datasets

When plentiful labeled data is available (>10,000 examples), updating every weight extracts more signal than a static body, leading to measurable gains in precision, recall, and F1 scores.

## Critical Trade-offs: Memory, Speed, and Stability

| Aspect | Frozen Body (Head-Only) | Unfrozen Body (Full Fine-Tuning) |
|--------|-------------------------|----------------------------------|
| **GPU RAM** | Minimal (head parameters only) | Requires full model + gradients (often >50 GB for 7B models) |
| **Training Time** | Short (few epochs) | Longer (more epochs, larger batches) |
| **Catastrophic Forgetting** | Low risk—body stays unchanged | High risk—body may lose pre-training knowledge if over-trained |
| **Accuracy Ceiling** | Modest baseline | Potentially higher on domain-shifted tasks |

## Implementation in the Curriculum

The `rohitg00/ai-engineering-from-scratch` repository explicitly implements these regimes through a unified training interface.

### The `freeze_body` Toggle

In [`phases/19-capstone-projects/38-classifier-finetuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/38-classifier-finetuning/docs/en.md), the curriculum provides a single `train_classifier` function that accepts a `freeze_body` boolean flag. When enabled, the implementation iterates through named parameters and sets `requires_grad=False` on all layers except the classifier, ensuring the optimizer "sees only the head and the body stays frozen."

### Key Source Files

- **[`phases/19-capstone-projects/38-classifier-finetuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/38-classifier-finetuning/docs/en.md)** – Documents the `freeze_body` toggle and explains when to apply each regime
- **[`phases/19-capstone-projects/38-classifier-finetuning/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/38-classifier-finetuning/code/main.py)** – Contains the concrete training loop implementation used in the lesson
- **[`phases/11-llm-engineering/08-fine-tuning-lora/outputs/skill-fine-tuning-guide.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/08-fine-tuning-lora/outputs/skill-fine-tuning-guide.md)** – Outlines parameter-efficient alternatives (LoRA) versus full fine-tuning
- **[`phases/19-capstone-projects/39-instruction-tuning-sft/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/39-instruction-tuning-sft/docs/en.md)** – Discusses supervised fine-tuning scenarios where the body typically remains unfrozen

## Practical Code Examples

The following PyTorch snippets mirror the pattern used in the **Classifier Fine-tuning** lesson, demonstrating both frozen and unfrozen approaches:

```python
import torch
import torch.nn as nn
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_name = "facebook/opt-350m"
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# ------------------------------------------------------------------

# 1️⃣  Frozen-body (head-only) fine-tuning

# ------------------------------------------------------------------

# Freeze all parameters except the classification head

for name, param in model.named_parameters():
    if "classifier" not in name:
        param.requires_grad = False

optimizer = torch.optim.AdamW(filter(lambda p: p.requires_grad, model.parameters()), lr=5e-5)

# Training loop (simplified)

for batch in dataloader:
    inputs = tokenizer(batch["text"], padding=True, truncation=True, return_tensors="pt")
    labels = batch["label"]
    outputs = model(**inputs, labels=labels)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

# ------------------------------------------------------------------

# 2️⃣  Unfrozen (full) fine-tuning

# ------------------------------------------------------------------

# Enable gradients for every parameter

for param in model.parameters():
    param.requires_grad = True

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)

# Same training loop as above but now the whole model is updated

```

## When to Choose Each Regime

| Scenario | Recommended Approach |
|----------|---------------------|
| **Few (< 1,000) labeled examples** | Freeze the body, train only the head |
| **Large (> 10,000) domain-shifted dataset** | Unfreeze the body for full fine-tuning |
| **Budget-constrained inference environments** | Freeze the body to minimize deployment costs |
| **Need to retain zero-shot capabilities** | Freeze the body or partially unfreeze only the last block |

## Summary

- **Frozen-body fine-tuning** sets `requires_grad=False` on base layers, optimizing only the classification head to save memory and prevent overfitting on small datasets.
- **Unfrozen fine-tuning** enables gradients across all parameters, maximizing accuracy for domain-shifted tasks but requiring substantial GPU resources.
- The `rohitg00/ai-engineering-from-scratch` curriculum implements these approaches via a `freeze_body` toggle in [`phases/19-capstone-projects/38-classifier-finetuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/38-classifier-finetuning/docs/en.md).
- **Catastrophic forgetting** poses minimal risk with frozen layers but becomes a significant concern when fully fine-tuning on specialized domains.
- Choose head-only training for limited data and resource constraints; select full fine-tuning when abundant data and compute are available.

## Frequently Asked Questions

### What happens to GPU memory when freezing layers in transfer learning?

When you freeze the model body, the optimizer only tracks gradients for the unfrozen head parameters. This reduces GPU memory consumption by approximately an order of magnitude compared to full fine-tuning, which must store gradients for every parameter in the architecture (often exceeding 50 GB for 7B parameter models).

### Can I unfreeze layers after initially training the head only?

Yes, this progressive unfreezing strategy is common in advanced implementations. You can first train the head with frozen layers to establish a baseline, then gradually unfreeze deeper layers (starting from the final transformer blocks) to allow domain adaptation while maintaining training stability. The curriculum's `train_classifier` function supports this by allowing dynamic toggling of the `freeze_body` flag between training phases.

### What is catastrophic forgetting in the context of fine-tuning?

**Catastrophic forgetting** occurs when a model loses previously learned general knowledge during fine-tuning on a specific task. When layers remain frozen, the risk is eliminated because the pre-trained weights never change. However, with full fine-tuning (unfrozen layers), the model may overwrite its broad linguistic or visual capabilities if trained too aggressively on a narrow dataset, reducing its effectiveness on out-of-distribution examples.

### Where is the freeze_body parameter implemented in the repository?

The `freeze_body` toggle is documented in [`phases/19-capstone-projects/38-classifier-finetuning/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/38-classifier-finetuning/docs/en.md) and implemented in the accompanying `train_classifier` function within [`phases/19-capstone-projects/38-classifier-finetuning/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/38-classifier-finetuning/code/main.py). This parameter controls whether the training loop applies `requires_grad=False` to body parameters, effectively switching between head-only and full fine-tuning regimes.