# How to Set the Vocabulary Size to Be Divisible by Any Number in Nanotron

> Learn how to set vocabulary size divisible by any number in Nanotron by configuring make_vocab_size_divisible_by. This pads embeddings for seamless tensor parallelism.

- Repository: [Hugging Face/nanotron](https://github.com/huggingface/nanotron)
- Tags: how-to-guide
- Published: 2026-03-03

---

**To set the vocabulary size to be divisible by a specific number in Nanotron, configure the `make_vocab_size_divisible_by` field in your model configuration, which automatically pads the embedding matrix to a multiple of that value times the tensor-parallel world size.**

When training large language models with the Hugging Face Nanotron framework, aligning the **vocabulary size** with hardware constraints prevents inefficient memory access patterns during distributed training. Nanotron automates this alignment through a configuration-driven padding mechanism defined in the source code, allowing you to enforce divisibility constraints without modifying model architectures manually.

## How Nanotron Calculates the Padded Vocabulary Size

Nanotron determines the final vocabulary size using a two-factor divisibility rule. The system computes the target padding as the smallest multiple of `tp_world_size × make_vocab_size_divisible_by` that is greater than or equal to your original vocabulary size.

The core logic resides in [`src/nanotron/helpers.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/helpers.py) inside the private helper `_vocab_size_with_padding` (lines 52–64). This function accepts the original `vocab_size`, the tensor-parallel world size, and your configured divisor. If the original size does not satisfy the divisibility constraint, Nanotron logs a warning and appends dummy tokens to reach the nearest valid multiple. The padded value is then stored in `self.model_config.vocab_size` and propagated to all embedding and output layers.

## Configuring the Divisibility Parameter

The `make_vocab_size_divisible_by` parameter is defined in [`src/nanotron/config/config.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/config/config.py) within the `ModelArgs` class (lines 12–14). It defaults to `1`, imposing no additional constraints beyond the tensor-parallel layout.

### YAML Configuration (Recommended)

Define the parameter in your training configuration file to ensure the vocabulary size is divisible by your target number. When training with a single tensor-parallel rank (`tp_world_size = 1`), setting this value to **N** ensures the final size is a multiple of **N**.

```yaml

# examples/config_tiny_llama.yaml

model:
  model_config:
    vocab_size: 32000          # Original size from tokenizer

  make_vocab_size_divisible_by: 8   # Force multiple of 8

tokens:
  sequence_length: 2048

```

When you launch training with [`run_train.py`](https://github.com/huggingface/nanotron/blob/main/run_train.py), Nanotron loads the tokenizer, retrieves the base `vocab_size` (e.g., 32000), and invokes `_vocab_size_with_padding`. If the original size is already divisible by `8 × tp_world_size`, no padding occurs; otherwise, the system adds dummy tokens to reach the next multiple.

### Command-Line Override

Override the YAML value without editing files by passing the parameter directly to the training script:

```bash
python run_train.py \
  --config-file examples/config_tiny_llama.yaml \
  --model.make_vocab_size_divisible_by 12 \
  --model.model_config.vocab_size 32768

```

This approach is useful for hyperparameter sweeps or testing different divisibility constraints against various hardware configurations.

## Verifying the Padding at Runtime

The padding is applied during model initialization in [`src/nanotron/trainer.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/trainer.py) (lines 88–92). To confirm the final vocabulary size, inspect the model configuration after the trainer initializes:

```python
from nanotron.trainer import Trainer

trainer = Trainer(config)
model = trainer.init_model()
print(f"Padded vocab size = {model.config.vocab_size}")

```

Alternatively, enable warning-level logging to see the exact padding count. Nanotron emits a message in the format:

```

[Vocab Size Padding] Padded vocab (size: 32761) with 39 dummy tokens (new size: 32800)

```

This confirms that 39 dummy tokens were added to reach the divisibility target.

## Handling Tensor Parallelism and Custom Tokenizers

When using tensor parallelism with a world size greater than 1, the effective divisor becomes the product of your configured value and `tp_world_size`. For example, if you set `make_vocab_size_divisible_by: 8` and train with `tp_world_size = 4`, the vocabulary must be divisible by 32.

If your custom tokenizer yields a non-standard vocabulary size such as 41000, Nanotron will pad it to 41024 (the next multiple of 32) by adding 24 dummy tokens. This ensures efficient sharding across tensor-parallel ranks without requiring you to manually resize embedding matrices or adjust the tokenizer.

## Summary

- **Configure** the `make_vocab_size_divisible_by` field in `ModelArgs` to control divisibility constraints.
- **Understand** that the final divisor equals `make_vocab_size_divisible_by × tp_world_size`.
- **Locate** the padding logic in [`src/nanotron/helpers.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/helpers.py) inside the `_vocab_size_with_padding` function.
- **Verify** padding through runtime logs or by inspecting `model.config.vocab_size` after trainer initialization.
- **Apply** settings via YAML, command-line arguments, or programmatic configuration objects.

## Frequently Asked Questions

### How do I disable vocabulary padding entirely?

Set `make_vocab_size_divisible_by` to `1` in your configuration. Since the default value is `1`, Nanotron will only enforce divisibility by the tensor-parallel world size itself, adding no extra padding beyond what the TP layout requires.

### Why is my final vocabulary size larger than my tokenizer's vocabulary?

Nanotron automatically pads the vocabulary to satisfy hardware alignment constraints. If you configure `make_vocab_size_divisible_by: 8` and use a tokenizer with 31999 tokens, Nanotron adds one dummy token to reach 32000. These extra tokens are ignored during training and do not affect the model's semantic output.

### Does this padding affect the output logits dimension?

Yes. The padded vocabulary size determines the final dimension of the language model head (output layer). While the padded tokens are technically part of the output space, they correspond to dummy embeddings that are never activated by valid input IDs, ensuring the model behavior remains identical to the unpadded version.

### Where can I see how many dummy tokens were added?

Check the training logs for the warning message emitted by `_vocab_size_with_padding` in [`src/nanotron/helpers.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/helpers.py). The message explicitly states the original size, the number of dummy tokens added, and the new padded size, allowing you to verify the exact padding count applied by the trainer.