# nanotron | Hugging Face | Knowledge Base | Instagit

Minimalistic large language model 3D-parallelism training

GitHub Stars: 2.6k

Repository: https://github.com/huggingface/nanotron

---

## Articles

### [How to Set the Vocabulary Size to Be Divisible by Any Number in Nanotron](/huggingface/nanotron/nanotron-set-vocabulary-size-divisible-by)

Learn how to set vocabulary size divisible by any number in Nanotron by configuring make_vocab_size_divisible_by. This pads embeddings for seamless tensor parallelism.

- Tags: how-to-guide
- Published: 2026-03-03

### [Model Initialization Methods in Nanotron: Random, MUP, and Checkpoint Loading Explained](/huggingface/nanotron/nanotron-model-initialization-methods)

Explore Nanotron model initialization: RandomInit, SpectralMupInit for width-stable MUP, and ExistingCheckpointInit for loading weights. Configure easily via init_method.

- Tags: deep-dive
- Published: 2026-03-03

### [How to Use CUDA Event-Based Timing for Debugging in Nanotron](/huggingface/nanotron/nanotron-cuda-event-based-timing-debugging)

Learn how to use Nanotron's CUDA event-based timing for efficient GPU debugging. Measure execution precisely without blocking CPU threads using torch.cuda.Event.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Set Up Multi-Stage Dataset Training in Nanotron](/huggingface/nanotron/nanotron-multi-stage-dataset-training-setup)

Learn how to set up multi-stage dataset training in Nanotron. Configure sequential training phases and automatic dataloader switching with the data_stages field for efficient model development.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Perform Generation with Temperature, Top-K, and Top-P Sampling in Nanotron](/huggingface/nanotron/nanotron-generation-temperature-top-k-top-p-sampling)

Master text generation in Nanotron. Learn to control output with temperature, top_k, and top_p sampling. Configure GenerationArgs and decode text for precise results. Enhance your NLP models today.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Configure Weight Decay with Parameter Exclusions in Nanotron](/huggingface/nanotron/nanotron-weight-decay-parameter-exclusions-configuration)

Configure weight decay with parameter exclusions in Nanotron. Easily exclude specific parameter types and maintain global decay rates for others. Optimize your model training effectively.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Use LightEval for Model Evaluation in Nanotron: A Complete Technical Guide](/huggingface/nanotron/nanotron-lighteval-model-evaluation-usage)

Learn to use LightEval for model evaluation in Nanotron. Automate LLM evaluation with checkpoints using LightEvalConfig, LightEvalRunner, and SLURM job submission.

- Tags: how-to-guide
- Published: 2026-03-03

### [Learning Rate Schedulers in Nanotron: Warm-Up and Decay Configuration Guide](/huggingface/nanotron/nanotron-available-learning-rate-schedulers)

Explore Nanotron's flexible learning rate schedulers. Configure linear/cosine decay with warm-up styles using LRSchedulerArgs for optimal training. Learn more now.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Implement Checkpointing with recompute_layer in Nanotron: A Complete Guide](/huggingface/nanotron/nanotron-checkpointing-recompute-layer-implementation)

Learn to implement checkpointing with recompute_layer in Nanotron to drastically reduce GPU memory by trading computation for memory. A complete guide.

- Tags: how-to-guide
- Published: 2026-03-03

### [Does Nanotron Support Flash Attention and Flex Attention?](/huggingface/nanotron/nanotron-flash-attention-flex-attention-support)

Yes Nanotron supports Flash Attention v1 v2 and Flex Attention. Learn how the unified registry architecture and config._attn_implementation enable runtime kernel swapping for faster inference.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Set Up Sequence Parallelism with Context Parallel Size in Nanotron](/huggingface/nanotron/nanotron-sequence-parallelism-context-parallel-size-setup)

Learn to set up sequence parallelism with context parallel size in Hugging Face Nanotron. Configure tp_mode and context_parallel_size for optimal distributed training.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Configure Tensor Parallel Mode (All-Reduce vs Reduce-Scatter) in Nanotron](/huggingface/nanotron/nanotron-tensor-parallel-mode-all-reduce-reduce-scatter-configuration)

Learn how to configure tensor parallel mode in Nanotron by choosing between all_reduce and reduce_scatter collectives. Optimize your distributed training performance.

- Tags: how-to-guide
- Published: 2026-03-03

### [What Is DoReMi and How to Use It in Nanotron](/huggingface/nanotron/nanotron-doremi-usage)

Learn about DoReMi, the Nanotron algorithm that auto-optimizes domain mixing. Discover how DoReMi dynamically adjusts training weights for efficient data domain focus and improved model performance.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Implement Parameter Tying Across Pipeline Stages in Nanotron](/huggingface/nanotron/nanotron-parameter-tying-pipeline-stages-implementation)

Learn to implement parameter tying across pipeline stages in Nanotron. This guide explains how to use tie_parameters and sync_tied_weights_gradients for efficient model parallelism. Reduce memory and boost performance.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Configure FP32 Gradient Accumulation in Nanotron](/huggingface/nanotron/nanotron-fp32-gradient-accumulation-configuration)

Unlock superior model training by enabling FP32 gradient accumulation in Nanotron. Simply set accumulate_grad_in_fp32: true in your optimizer config with zero_stage 1 or higher. Train with higher precision.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Use a Custom DataLoader with Nanotron: Implementation Guide](/huggingface/nanotron/nanotron-custom-dataloader-usage)

Learn how to use a custom dataloader with Nanotron. Implement a collator for specific batch dictionary formats and TensorPointers for efficient distributed training.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Resume Training from a Checkpoint in Nanotron](/huggingface/nanotron/nanotron-resume-training-from-checkpoint)

Easily resume training from a checkpoint in Nanotron. Set resume_checkpoint_path and run train.py to seamlessly continue your model training.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Configure S3 Checkpoint Upload with s5cmd in Nanotron](/huggingface/nanotron/nanotron-s3-checkpoint-upload-s5cmd-configuration)

Learn how to configure S3 checkpoint upload with s5cmd in Nanotron. Stream training checkpoints to Amazon S3 seamlessly using YAML configuration for efficient data management.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Implement Ring Attention for Long Sequences in Nanotron](/huggingface/nanotron/nanotron-ring-attention-long-sequences-implementation)

Implement ring attention for long sequences in Nanotron using ring_flash_attn_varlen_func Achieve efficient memory scaling for ultra-long sequences by sharding KV tensors across your GPU ring.

- Tags: how-to-guide
- Published: 2026-03-03

### [When to Use Spectral µTransfer Parametrization in Nanotron](/huggingface/nanotron/nanotron-spectral-mu-transfer-parametrization-usage)

Learn when to use Spectral µTransfer parametrization in Nanotron for deep transformer models. Achieve depth-independent scaling and hyperparameter transfer across model sizes with MUP.

- Tags: deep-dive
- Published: 2026-03-03

### [How to Configure AFAB vs 1F1B Pipeline Schedules in Nanotron](/huggingface/nanotron/nanotron-afab-vs-1f1b-pipeline-schedule-configuration)

Configure AFAB vs 1F1B pipeline schedules in Nanotron Nanotron using YAML, Python API, or CLI. Optimize your training flow with All-Forward-All-Backward or Interleaved-Forward-Interleaved-Backward execution.

- Tags: how-to-guide
- Published: 2026-03-03

### [How to Implement Mixture of Experts (MoE) with Expert Parallelism in Nanotron](/huggingface/nanotron/nanotron-moe-expert-parallelism-implementation)

Learn to implement Mixture of Experts MoE with expert parallelism in Nanotron. Configure expert parallelism, create parallel context, and utilize Qwen2MoELayer for efficient MoE implementation.

- Tags: how-to-guide
- Published: 2026-03-03

### [Memory Benefits of ZeRO-1 Optimizer in Nanotron: A Deep Dive](/huggingface/nanotron/nanotron-zero-1-optimizer-memory-benefits)

Discover ZeRO-1 optimizer memory benefits in Nanotron. Shard optimizer states to cut per-GPU memory usage significantly while keeping full parameter access for efficient model training.

- Tags: deep-dive
- Published: 2026-03-03

### [How to Configure 3D Parallelism (DP+TP+PP) in Hugging Face Nanotron](/huggingface/nanotron/how-to-configure-3d-parallelism-dp-tp-pp-in-nanotron)

Learn to configure 3D parallelism DP+TP+PP in Hugging Face Nanotron. Set YAML config and use torchrun for efficient distributed training with simple steps.

- Tags: how-to-guide
- Published: 2026-03-03