# Memory Benefits of ZeRO-1 Optimizer in Nanotron: A Deep Dive

> Discover ZeRO-1 optimizer memory benefits in Nanotron. Shard optimizer states to cut per-GPU memory usage significantly while keeping full parameter access for efficient model training.

- Repository: [Hugging Face/nanotron](https://github.com/huggingface/nanotron)
- Tags: deep-dive
- Published: 2026-03-03

---

**The ZeRO-1 optimizer in Nanotron reduces per-GPU memory usage by sharding optimizer states across the data-parallel dimension, cutting memory consumption roughly by the data-parallel world size while maintaining full model parameter access for training.**

Nanotron implements ZeRO-1 (Zero Redundancy Optimizer stage 1) to tackle the memory bottleneck that limits large-scale transformer training. By distributing optimizer states across data-parallel ranks rather than replicating them on every GPU, the framework enables training of significantly larger models or larger batch sizes on existing hardware. This article examines the specific memory optimization mechanisms implemented in [`src/nanotron/optim/zero.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/optim/zero.py) and their practical impact on training efficiency.

## How ZeRO-1 Reduces Memory in Nanotron

### Parameter Sharding Across Data-Parallel Ranks

The core memory benefit stems from **parameter-wise sharding** implemented in [`src/nanotron/optim/zero.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/optim/zero.py). Instead of each GPU maintaining a complete copy of every optimizer state (momentum buffers, variance estimates), the `ZeRODistributedOptimizer` class assigns each data-parallel rank responsibility for only a subset of parameters.

The implementation tracks this through `self.id_to_name`, which holds only the parameter names relevant to the current rank. This design reduces per-GPU optimizer-state memory approximately by the data-parallel world size. For example, with 8 data-parallel ranks, each GPU stores only 1/8th of the total optimizer states, transforming a 16 GB optimizer memory footprint into a 2 GB allocation per device.

### Lazy Allocation of Optimizer States

Nanotron implements **lazy allocation** to prevent the "double-allocation" problem that occurs when full-precision parameters and their optimizer states coexist temporarily. The optimizer states are created only after model parameters have been sharded across ranks, ensuring that initial memory allocation contains only the model weights.

The source code includes explicit memory logging at lines 195-196 in [`zero.py`](https://github.com/huggingface/nanotron/blob/main/zero.py) that reports GPU memory usage before and after this allocation phase. This allows users to verify the memory savings during initialization and debug potential out-of-memory errors during the sharding process.

## Checkpoint and Serialization Benefits

### Reduced Checkpoint Size

ZeRO-1 sharding significantly reduces checkpoint storage requirements and I/O overhead. When persisting training state, Nanotron saves only the local optimizer shard rather than full copies for every rank. This produces checkpoint files following the naming pattern `optimizer_pp-0-of-1_dp-0-of-2.pt`, where the `dp` component indicates the data-parallel shard index.

For a model with 10 billion parameters, this reduces per-checkpoint optimizer state storage from approximately 40 GB (Adam states in FP32) to roughly 5 GB per rank when using 8-way data parallelism. This reduction minimizes distributed filesystem contention and accelerates checkpoint rotation during long training runs.

### Shard Merging for Evaluation

Nanotron provides utilities to merge ZeRO-1 shards back into a complete optimizer state when required for evaluation or fine-tuning workflows. The merge operation, indicated by the log message `desc="Merging ZeRO-1's shards..."`, reconstructs the full state only when necessary rather than maintaining it throughout training.

This design choice preserves memory efficiency during the training loop while ensuring compatibility with downstream tasks that require complete model and optimizer states. The merge functionality resides in the serialization layer, separate from the critical training path in [`zero.py`](https://github.com/huggingface/nanotron/blob/main/zero.py).

## Implementation Details in Nanotron

Activating ZeRO-1 requires minimal configuration changes. Through the command-line interface in [`slurm_launcher.py`](https://github.com/huggingface/nanotron/blob/main/slurm_launcher.py), users specify `--zero 1` to enable stage 1 optimization:

```bash
python -m nanotron.slurm_launcher \
    --config ./examples/config_tiny_llama.py \
    --zero 1

```

For direct Python instantiation, the `ZeRODistributedOptimizer` class accepts the zero stage parameter explicitly:

```python
from nanotron.optim.zero import ZeRODistributedOptimizer

optimizer = ZeRODistributedOptimizer(
    model.parameters(),
    lr=1e-4,
    zero_stage=1,
)

```

The implementation automatically handles the parameter sharding logic, requiring no manual intervention to distribute optimizer states across ranks. Memory monitoring is built into the initialization process, with the following pattern available for debugging:

```python
import torch
import logging

# Memory logging pattern from zero.py lines 195-196

logger = logging.getLogger(__name__)
current_mem = torch.cuda.memory_allocated() / 1024**2
logger.info(f"[ZeRO sharding] Size of optimizer params per rank: {current_mem:.2f} MB")

```

## Summary

- **ZeRO-1 in Nanotron shards optimizer states across data-parallel ranks**, reducing per-GPU memory usage by approximately the DP world size.
- **Lazy allocation prevents double-memory usage** during initialization by creating optimizer states only after parameter sharding completes.
- **Checkpoint sizes shrink proportionally** to the sharding factor, reducing I/O overhead and storage requirements for distributed training runs.
- **Shard merging utilities** enable full state reconstruction for evaluation without compromising training memory efficiency.
- **Implementation resides in [`src/nanotron/optim/zero.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/optim/zero.py)**, with activation via `--zero 1` CLI flag or direct `ZeRODistributedOptimizer` instantiation.

## Frequently Asked Questions

### How much memory does ZeRO-1 actually save in Nanotron?

ZeRO-1 reduces per-GPU optimizer state memory by a factor equal to the data-parallel world size. For example, with 8-way data parallelism training a model using Adam optimizer (which stores two FP32 states per parameter), the optimizer memory per GPU drops from approximately 8 bytes per parameter to 1 byte per parameter. This often translates to saving tens of gigabytes per GPU for billion-parameter models.

### Does ZeRO-1 affect training speed or convergence?

ZeRO-1 introduces minimal communication overhead compared to standard data-parallel training because it only requires gathering parameters during the forward and backward passes, not the optimizer states. The convergence behavior remains identical to standard Adam or other optimizers since the mathematical operations are equivalent—only the storage location changes. The implementation in [`src/nanotron/optim/zero.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/optim/zero.py) handles the necessary communication transparently.

### When should I use ZeRO-1 versus higher ZeRO stages in Nanotron?

Use ZeRO-1 when optimizer states constitute the primary memory bottleneck but you have sufficient memory to store the full model parameters and gradients on each GPU. This is the most common scenario for models up to several billion parameters. Consider ZeRO-2 or ZeRO-3 only when activations or model parameters themselves exceed single-GPU memory limits, as these stages introduce additional communication costs that ZeRO-1 avoids.

### How do I verify that ZeRO-1 sharding is working correctly in my training run?

Monitor the logs for messages from [`src/nanotron/optim/zero.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/optim/zero.py) indicating memory allocation before and after optimizer initialization. The log entries showing `[ZeRO sharding] Size of optimizer params per rank` confirm that sharding occurred. Additionally, check your checkpoint directory for files named with the pattern `optimizer_pp-*_dp-*-of-*.pt`, which indicates the optimizer state was saved in sharded format rather than as a full replica.