Memory Benefits of ZeRO-1 Optimizer in Nanotron: A Deep Dive

The ZeRO-1 optimizer in Nanotron reduces per-GPU memory usage by sharding optimizer states across the data-parallel dimension, cutting memory consumption roughly by the data-parallel world size while maintaining full model parameter access for training.

Nanotron implements ZeRO-1 (Zero Redundancy Optimizer stage 1) to tackle the memory bottleneck that limits large-scale transformer training. By distributing optimizer states across data-parallel ranks rather than replicating them on every GPU, the framework enables training of significantly larger models or larger batch sizes on existing hardware. This article examines the specific memory optimization mechanisms implemented in src/nanotron/optim/zero.py and their practical impact on training efficiency.

How ZeRO-1 Reduces Memory in Nanotron

Parameter Sharding Across Data-Parallel Ranks

The core memory benefit stems from parameter-wise sharding implemented in src/nanotron/optim/zero.py. Instead of each GPU maintaining a complete copy of every optimizer state (momentum buffers, variance estimates), the ZeRODistributedOptimizer class assigns each data-parallel rank responsibility for only a subset of parameters.

The implementation tracks this through self.id_to_name, which holds only the parameter names relevant to the current rank. This design reduces per-GPU optimizer-state memory approximately by the data-parallel world size. For example, with 8 data-parallel ranks, each GPU stores only 1/8th of the total optimizer states, transforming a 16 GB optimizer memory footprint into a 2 GB allocation per device.

Lazy Allocation of Optimizer States

Nanotron implements lazy allocation to prevent the "double-allocation" problem that occurs when full-precision parameters and their optimizer states coexist temporarily. The optimizer states are created only after model parameters have been sharded across ranks, ensuring that initial memory allocation contains only the model weights.

The source code includes explicit memory logging at lines 195-196 in zero.py that reports GPU memory usage before and after this allocation phase. This allows users to verify the memory savings during initialization and debug potential out-of-memory errors during the sharding process.

Checkpoint and Serialization Benefits

Reduced Checkpoint Size

ZeRO-1 sharding significantly reduces checkpoint storage requirements and I/O overhead. When persisting training state, Nanotron saves only the local optimizer shard rather than full copies for every rank. This produces checkpoint files following the naming pattern optimizer_pp-0-of-1_dp-0-of-2.pt, where the dp component indicates the data-parallel shard index.

For a model with 10 billion parameters, this reduces per-checkpoint optimizer state storage from approximately 40 GB (Adam states in FP32) to roughly 5 GB per rank when using 8-way data parallelism. This reduction minimizes distributed filesystem contention and accelerates checkpoint rotation during long training runs.

Shard Merging for Evaluation

Nanotron provides utilities to merge ZeRO-1 shards back into a complete optimizer state when required for evaluation or fine-tuning workflows. The merge operation, indicated by the log message desc="Merging ZeRO-1's shards...", reconstructs the full state only when necessary rather than maintaining it throughout training.

This design choice preserves memory efficiency during the training loop while ensuring compatibility with downstream tasks that require complete model and optimizer states. The merge functionality resides in the serialization layer, separate from the critical training path in zero.py.

Implementation Details in Nanotron

Activating ZeRO-1 requires minimal configuration changes. Through the command-line interface in slurm_launcher.py, users specify --zero 1 to enable stage 1 optimization:

python -m nanotron.slurm_launcher \
    --config ./examples/config_tiny_llama.py \
    --zero 1

For direct Python instantiation, the ZeRODistributedOptimizer class accepts the zero stage parameter explicitly:

from nanotron.optim.zero import ZeRODistributedOptimizer

optimizer = ZeRODistributedOptimizer(
    model.parameters(),
    lr=1e-4,
    zero_stage=1,
)

The implementation automatically handles the parameter sharding logic, requiring no manual intervention to distribute optimizer states across ranks. Memory monitoring is built into the initialization process, with the following pattern available for debugging:

import torch
import logging

# Memory logging pattern from zero.py lines 195-196

logger = logging.getLogger(__name__)
current_mem = torch.cuda.memory_allocated() / 1024**2
logger.info(f"[ZeRO sharding] Size of optimizer params per rank: {current_mem:.2f} MB")

Summary

• ZeRO-1 in Nanotron shards optimizer states across data-parallel ranks, reducing per-GPU memory usage by approximately the DP world size.
• Lazy allocation prevents double-memory usage during initialization by creating optimizer states only after parameter sharding completes.
• Checkpoint sizes shrink proportionally to the sharding factor, reducing I/O overhead and storage requirements for distributed training runs.
• Shard merging utilities enable full state reconstruction for evaluation without compromising training memory efficiency.
• Implementation resides in src/nanotron/optim/zero.py, with activation via --zero 1 CLI flag or direct ZeRODistributedOptimizer instantiation.

Frequently Asked Questions

How much memory does ZeRO-1 actually save in Nanotron?

ZeRO-1 reduces per-GPU optimizer state memory by a factor equal to the data-parallel world size. For example, with 8-way data parallelism training a model using Adam optimizer (which stores two FP32 states per parameter), the optimizer memory per GPU drops from approximately 8 bytes per parameter to 1 byte per parameter. This often translates to saving tens of gigabytes per GPU for billion-parameter models.

Does ZeRO-1 affect training speed or convergence?

ZeRO-1 introduces minimal communication overhead compared to standard data-parallel training because it only requires gathering parameters during the forward and backward passes, not the optimizer states. The convergence behavior remains identical to standard Adam or other optimizers since the mathematical operations are equivalent—only the storage location changes. The implementation in src/nanotron/optim/zero.py handles the necessary communication transparently.

When should I use ZeRO-1 versus higher ZeRO stages in Nanotron?

Use ZeRO-1 when optimizer states constitute the primary memory bottleneck but you have sufficient memory to store the full model parameters and gradients on each GPU. This is the most common scenario for models up to several billion parameters. Consider ZeRO-2 or ZeRO-3 only when activations or model parameters themselves exceed single-GPU memory limits, as these stages introduce additional communication costs that ZeRO-1 avoids.

How do I verify that ZeRO-1 sharding is working correctly in my training run?

Monitor the logs for messages from src/nanotron/optim/zero.py indicating memory allocation before and after optimizer initialization. The log entries showing [ZeRO sharding] Size of optimizer params per rank confirm that sharding occurred. Additionally, check your checkpoint directory for files named with the pattern optimizer_pp-*_dp-*-of-*.pt, which indicates the optimizer state was saved in sharded format rather than as a full replica.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}