# When to Use Spectral µTransfer Parametrization in Nanotron

> Learn when to use Spectral µTransfer parametrization in Nanotron for deep transformer models. Achieve depth-independent scaling and hyperparameter transfer across model sizes with MUP.

- Repository: [Hugging Face/nanotron](https://github.com/huggingface/nanotron)
- Tags: deep-dive
- Published: 2026-03-03

---

**Use Spectral µTransfer (MUP) parametrization in Nanotron when training deep transformer models where you need depth-independent scaling of weight updates and hyperparameter transfer across model sizes.**

Nanotron is Hugging Face's lightweight library for pretraining Large Language Models (LLMs). When scaling transformer architectures to hundreds of layers, **Spectral µTransfer parametrization in Nanotron** provides a principled initialization strategy that preserves feature-learning dynamics regardless of model depth, based on the "Spectral Condition for Feature Learning" research by Yang et al. (2023).

## What is Spectral µTransfer and When to Apply It

Nanotron supports two distinct initialization strategies for transformer-style models, each serving different scaling requirements.

### Standard Initialization vs. Spectral MUP

- **Standard (Random) Init**: Implemented via the `RandomInit` class in [`src/nanotron/config/models_config.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/config/models_config.py). This approach uses a fixed standard deviation (typically `std=0.02`) and is appropriate for typical training scenarios where you manually tune learning-rate schedules and the model depth is modest.

- **Spectral MUP Init**: Implemented via `SpectralMupInit(use_mup=True)` and the `SpectralMupParametrizator` class in [`src/nanotron/scaling/parametrization.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/scaling/parametrization.py). This method computes a *spectral* standard deviation that scales inversely with layer width, ensuring that weight updates remain stable regardless of network depth.

### When to Enable Spectral MUP

Activate **Spectral µTransfer parametrization** when:

1. You are training LLMs with many transformer layers (very deep architectures).
2. You require the same learning-rate hyperparameters to work when scaling model depth or width (hyperparameter transfer).
3. You are conducting research based on the Spectral Condition for Feature Learning, which recommends parametrizing weights with a spectral standard deviation of `(1 / sqrt(fan_in)) * min(1, sqrt(fan_out / fan_in))`.

## How Nanotron Implements Spectral MUP

The implementation propagates the MUP flag through four distinct layers of the codebase, from configuration to weight initialization.

### Configuration Layer

The entry point is the `ModelArgs` dataclass defined in [`src/nanotron/config/models_config.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/config/models_config.py) (lines 18-26). To enable MUP, instantiate:

```python
from nanotron.config import ModelArgs, SpectralMupInit

model_args = ModelArgs(
    init_method=SpectralMupInit(use_mup=True),
    model_config=your_config,
)

```

### Flag Propagation

During `Config` construction, Nanotron detects whether the initialization method is a `SpectralMupInit` and records this in the model configuration via the `_is_using_mup` flag. This logic resides in [`src/nanotron/config/config.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/config/config.py) at line 324. This boolean flag subsequently drives all MUP-specific behavior throughout the training pipeline.

### The SpectralMupParametrizator Class

When `ParametrizationMethod.SPECTRAL_MUP` is selected, Nanotron instantiates `SpectralMupParametrizator` from [`src/nanotron/scaling/parametrization.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/scaling/parametrization.py) (lines 25-66). The core method `_parametrize_mup_weight` implements the spectral scaling formula:

```python
std = (1 / sqrt(fan_in)) * min(1, sqrt(fan_out / fan_in))
init.normal_(data, mean=0.0, std=std)

```

This guarantees that the variance of forward activations stays stable regardless of network depth, following the Spectral Condition for Feature Learning.

### Architecture-Specific Adjustments

Modules query `self.is_using_mup` to adjust behavior dynamically. In [`src/nanotron/models/llama.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/models/llama.py) (lines 262-289), the attention mechanism checks this flag to set the softmax scaling factor. When MUP is active, the scale becomes `1 / query_dim` instead of the standard `1 / sqrt(d_k)`, reflecting the MUP-adjusted geometry required for stable gradients in deep models.

## Configuring Spectral MUP in Your Training Pipeline

To apply Spectral µTransfer parametrization in practice, you must set both the initialization dataclass and the parametrization method enum.

### Basic Setup with ModelArgs

```python
from nanotron.config import ModelArgs, SpectralMupInit
from nanotron.scaling.parametrization import ParametrizationMethod
from nanotron.models.llama import LlamaModel
from nanotron.parallel import ParallelContext

# Configure with Spectral MUP

model_args = ModelArgs(
    init_method=SpectralMupInit(use_mup=True),
    model_config=TINY_LLAMA_CONFIG,
)

# Initialize with the correct parametrization method

llama.init_model_randomly(
    config=model_args,
    init_method=ParametrizationMethod.SPECTRAL_MUP,
)

```

This pattern is validated in the test suite at [`tests/test_parametrization.py`](https://github.com/huggingface/nanotron/blob/main/tests/test_parametrization.py) (lines 22-23), where a tiny Llama model is initialized and the expected standard deviations are verified against the spectral formula.

### Switching Between Initialization Methods

You can toggle between strategies by changing the `init_method` and `ParametrizationMethod` enum:

```python

# Standard initialization

standard_args = ModelArgs(
    init_method=RandomInit(std=0.02),
    model_config=TINY_LLAMA_CONFIG,
)
llama.init_model_randomly(
    config=standard_args,
    init_method=ParametrizationMethod.STANDARD,
)

# Spectral MUP initialization (same model architecture, different scaling)

mup_args = ModelArgs(
    init_method=SpectralMupInit(use_mup=True),
    model_config=TINY_LLAMA_CONFIG,
)
llama.init_model_randomly(
    config=mup_args,
    init_method=ParametrizationMethod.SPECTRAL_MUP,
)

```

## Summary

- Use **Spectral µTransfer parametrization** when training deep transformers where depth-independent hyperparameter transfer is required.
- Enable it by setting `SpectralMupInit(use_mup=True)` in `ModelArgs` and selecting `ParametrizationMethod.SPECTRAL_MUP`.
- The implementation resides in [`src/nanotron/scaling/parametrization.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/scaling/parametrization.py), utilizing the spectral standard deviation formula `(1 / sqrt(fan_in)) * min(1, sqrt(fan_out / fan_in))`.
- Architecture-specific adjustments (like attention scaling in [`src/nanotron/models/llama.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/models/llama.py)) automatically activate when `is_using_mup` is detected.
- Unit tests in [`tests/test_parametrization.py`](https://github.com/huggingface/nanotron/blob/main/tests/test_parametrization.py) verify the correct statistical properties of the initialized weights.

## Frequently Asked Questions

### What is the primary difference between Standard Init and Spectral MUP in Nanotron?

Standard initialization uses a fixed standard deviation across all layers, requiring manual learning-rate tuning as depth increases. Spectral MUP uses a data-dependent spectral standard deviation that scales inversely with layer width, enabling the same learning rate to work across different model depths and widths without retuning.

### How does Spectral MUP affect attention scaling in Nanotron models?

According to [`src/nanotron/models/llama.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/models/llama.py) (lines 262-289), when `is_using_mup` is `True`, the attention softmax scaling changes from `1 / sqrt(d_k)` to `1 / query_dim`. This adjustment reflects the µTransfer geometry and ensures stable gradient flow in deeply stacked transformer layers.

### Can I apply Spectral MUP to a model that has already been initialized with Standard Init?

No. Spectral MUP must be applied at model construction time via `init_model_randomly()` with `ParametrizationMethod.SPECTRAL_MUP`. Switching initialization strategies requires reinitializing the model weights from scratch, as the spectral scaling formula fundamentally alters the initial weight distributions and the subsequent learning dynamics.

### Which configuration file controls the MUP flag propagation in Nanotron?

The `_is_using_mup` flag is set in [`src/nanotron/config/config.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/config/config.py) at line 324 during `Config` construction. This flag is derived from checking whether the `init_method` is an instance of `SpectralMupInit`, and it propagates to all model components that require MUP-aware behavior, such as the attention mechanisms in [`src/nanotron/models/llama.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/models/llama.py).