# How nanochat Combines Muon and AdamW in a Single Optimizer

> Discover how nanochat combines Muon and AdamW optimizers by partitioning parameters to accelerate training. Learn about its unique approach to tensor optimization for deep learning models.

- Repository: [Andrej/nanochat](https://github.com/karpathy/nanochat)
- Tags: internals
- Published: 2026-03-10

---

**nanochat merges Muon and AdamW by partitioning parameters into distinct groups—applying AdamW to 0-D and 1-D tensors while using Muon for 2-D matrices—then dispatching to specialized fused kernels within a unified `step()` method.**

The `karpathy/nanochat` repository implements a hybrid optimization strategy that leverages the complementary strengths of both Muon and AdamW. This combined approach assigns each algorithm to the parameter shapes where it excels most, delivering efficient training for large language models. Understanding how nanochat's optimizer combines Muon and AdamW requires examining its parameter grouping logic, fused kernel implementation, and distributed training support.

## Parameter Grouping Strategy

nanochat's optimization begins with intelligent parameter partitioning. The model's `setup_optimizer` method categorizes every parameter into one of two groups based on dimensionality: tensors with fewer than two dimensions are flagged for AdamW, while 2-D matrices are designated for Muon processing.

This grouping occurs in the training scripts, where the model factory method constructs the optimizer instance. The method accepts separate learning rate arguments for each parameter type, allowing fine-grained control over the optimization dynamics. Referencing the source in [[`scripts/base_train.py`](https://github.com/karpathy/nanochat/blob/main/scripts/base_train.py)](https://github.com/karpathy/nanochat/blob/master/scripts/base_train.py#L306-L315), the setup call passes distinct learning rates for embeddings, scalars, and matrices, which the optimizer uses to configure its internal parameter groups.

## The MuonAdamW Class Architecture

At the core of nanochat's optimization engine sits the [`MuonAdamW`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L152-L164) class, a custom PyTorch optimizer that orchestrates the dual-algorithm approach. Unlike traditional optimizers that apply a single update rule universally, this class maintains separate state management and step functions for each parameter category.

The class inherits from `torch.optim.Optimizer` and overrides the `step()` method to implement conditional dispatch. When invoked, it iterates through `self.param_groups` and routes each group to either `_step_adamw` or `_step_muon` based on the group's `kind` attribute. This architectural decision keeps the two optimization paths completely independent while presenting a unified interface to the training loop.

### AdamW Step Implementation

The AdamW path handles all non-matrix parameters including embeddings, biases, and layer normalization statistics. The [`_step_adamw`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L194-L227) method initializes per-parameter state dictionaries containing first and second moment estimates (`exp_avg` and `exp_avg_sq`).

To maximize throughput, the implementation populates several 0-D CPU tensors with current hyperparameters—step count, learning rate, betas, epsilon, and weight decay—then invokes the [`adamw_step_fused`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L20-L50) kernel. This fused operation, compiled with `@torch.compile`, executes the entire AdamW update computation in a single graph, eliminating Python interpreter overhead during the parameter update phase.

### Muon Step Implementation

For 2-D weight matrices, nanochat employs the Muon optimizer via the [`_step_muon`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L329-L382) method. This implementation handles the orthogonalization-based updates that characterize Muon's approach to optimization. The method aggregates gradients and parameters from all matrix-shaped tensors in the group into stacked tensors to enable vectorized processing.

The Muon step maintains specialized momentum buffers and performs Polar-Express orthogonalization along with variance reduction techniques. After preparing the 0-D hyperparameter tensors for Muon-specific settings, it calls the [`muon_step_fused`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L90-L147) kernel. This compiled kernel applies Nesterov momentum, performs the orthogonalization, and applies cautious weight decay before writing the updated values back to the original parameter tensors.

## Fused Kernel Optimization

Both optimization paths rely on custom fused kernels to minimize dispatch overhead. The [`adamw_step_fused`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L20-L50) and [`muon_step_fused`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L90-L147) functions are decorated with `@torch.compile`, allowing PyTorch to generate optimized machine code for the entire update computation.

This design choice proves critical for training efficiency, as it prevents the Python global interpreter lock from becoming a bottleneck during parameter updates. By compiling the inner loops into optimized kernels, nanochat achieves performance comparable to manually written CUDA operations while maintaining the flexibility of Pythonic optimizer implementations.

## Distributed Training Support

When training across multiple GPUs with Distributed Data Parallel (DDP), nanochat substitutes `MuonAdamW` with [`DistMuonAdamW`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L294-L334). This subclass implements a three-phase asynchronous communication pattern that overlaps gradient reduction with computation.

The distributed variant maintains the same dual-algorithm structure but adds specialized communication hooks. These hooks coordinate all-reduce operations for both AdamW and Muon parameter groups while preserving the independent update semantics of the single-GPU implementation. The result allows linear scaling of the hybrid optimization strategy across many accelerators without modifying the underlying training loop code.

## Practical Usage Examples

Instantiating the combined optimizer requires a single call to the model's setup method. The following pattern appears throughout nanochat's training scripts:

```python

# Initialize the hybrid optimizer with separate learning rates

optimizer = model.setup_optimizer(
    unembedding_lr=args.unembedding_lr * batch_lr_scale,
    embedding_lr=args.embedding_lr * batch_lr_scale,
    scalar_lr=args.scalar_lr * batch_lr_scale,
    matrix_lr=args.matrix_lr * batch_lr_scale,
    weight_decay=weight_decay_scaled,
)

```

Within the training loop, usage remains identical to standard PyTorch optimizers:

```python
for iteration, (inputs, targets) in enumerate(train_loader):
    optimizer.zero_grad(set_to_none=True)
    outputs = model(inputs)
    loss = compute_loss(outputs, targets)
    loss.backward()
    optimizer.step()  # Dispatches to AdamW or Muon based on parameter shape

```

For distributed configurations, the model factory automatically returns a `DistMuonAdamW` instance when `args.ddp` is enabled, requiring no changes to the optimization loop:

```python
if args.ddp:
    optimizer = model.setup_optimizer(...)  # Returns DistMuonAdamW

else:
    optimizer = model.setup_optimizer(...)  # Returns MuonAdamW

```

## Summary

- nanochat partitions parameters into two distinct groups: 0-D/1-D tensors use AdamW while 2-D matrices use Muon.
- The [`MuonAdamW`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L152-L164) class dispatches to specialized step methods based on the `kind` attribute of each parameter group.
- Fused kernels [`adamw_step_fused`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L20-L50) and [`muon_step_fused`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L90-L147) compile the update logic for maximum performance.
- The [`DistMuonAdamW`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L294-L334) variant extends this architecture to multi-GPU training with asynchronous communication.

## Frequently Asked Questions

### Which parameters does Muon optimize versus AdamW in nanochat?

Muon exclusively handles 2-D matrix-shaped weight parameters, such as the weight matrices in linear and attention layers. AdamW manages all other parameters including 1-D embeddings, biases, layer normalization statistics, and scalar values, ensuring stable optimization for non-matrix tensors.

### Why does nanochat use fused kernels for the optimizer steps?

The fused kernels eliminate Python interpreter overhead by compiling the entire parameter update computation into optimized machine code using `@torch.compile`. This approach maximizes GPU utilization and training throughput, particularly important when updating billions of parameters in large language models.

### How does distributed training work with the combined optimizer?

The [`DistMuonAdamW`](https://github.com/karpathy/nanochat/blob/master/nanochat/optim.py#L294-L334) class extends the base optimizer with a three-phase asynchronous communication pattern that handles gradient all-reduction across GPUs. It preserves the dual-algorithm structure while overlapping communication with computation, enabling efficient scaling without code changes to the training loop.

### Can I adjust learning rates independently for Muon and AdamW parameters?

Yes, the `setup_optimizer` method accepts separate learning rate arguments including `matrix_lr` for Muon-optimized weights and distinct rates for embeddings, scalars, and unembedding layers. This design allows precise control over the optimization dynamics for different parameter types within the same training run.