How to Set the Vocabulary Size to Be Divisible by Any Number in Nanotron

To set the vocabulary size to be divisible by a specific number in Nanotron, configure the make_vocab_size_divisible_by field in your model configuration, which automatically pads the embedding matrix to a multiple of that value times the tensor-parallel world size.

When training large language models with the Hugging Face Nanotron framework, aligning the vocabulary size with hardware constraints prevents inefficient memory access patterns during distributed training. Nanotron automates this alignment through a configuration-driven padding mechanism defined in the source code, allowing you to enforce divisibility constraints without modifying model architectures manually.

How Nanotron Calculates the Padded Vocabulary Size

Nanotron determines the final vocabulary size using a two-factor divisibility rule. The system computes the target padding as the smallest multiple of tp_world_size × make_vocab_size_divisible_by that is greater than or equal to your original vocabulary size.

The core logic resides in src/nanotron/helpers.py inside the private helper _vocab_size_with_padding (lines 52–64). This function accepts the original vocab_size, the tensor-parallel world size, and your configured divisor. If the original size does not satisfy the divisibility constraint, Nanotron logs a warning and appends dummy tokens to reach the nearest valid multiple. The padded value is then stored in self.model_config.vocab_size and propagated to all embedding and output layers.

Configuring the Divisibility Parameter

The make_vocab_size_divisible_by parameter is defined in src/nanotron/config/config.py within the ModelArgs class (lines 12–14). It defaults to 1, imposing no additional constraints beyond the tensor-parallel layout.

YAML Configuration (Recommended)

Define the parameter in your training configuration file to ensure the vocabulary size is divisible by your target number. When training with a single tensor-parallel rank (tp_world_size = 1), setting this value to N ensures the final size is a multiple of N.

# examples/config_tiny_llama.yaml

model:
  model_config:
    vocab_size: 32000          # Original size from tokenizer

  make_vocab_size_divisible_by: 8   # Force multiple of 8

tokens:
  sequence_length: 2048

When you launch training with run_train.py, Nanotron loads the tokenizer, retrieves the base vocab_size (e.g., 32000), and invokes _vocab_size_with_padding. If the original size is already divisible by 8 × tp_world_size, no padding occurs; otherwise, the system adds dummy tokens to reach the next multiple.

Command-Line Override

Override the YAML value without editing files by passing the parameter directly to the training script:

python run_train.py \
  --config-file examples/config_tiny_llama.yaml \
  --model.make_vocab_size_divisible_by 12 \
  --model.model_config.vocab_size 32768

This approach is useful for hyperparameter sweeps or testing different divisibility constraints against various hardware configurations.

Verifying the Padding at Runtime

The padding is applied during model initialization in src/nanotron/trainer.py (lines 88–92). To confirm the final vocabulary size, inspect the model configuration after the trainer initializes:

from nanotron.trainer import Trainer

trainer = Trainer(config)
model = trainer.init_model()
print(f"Padded vocab size = {model.config.vocab_size}")

Alternatively, enable warning-level logging to see the exact padding count. Nanotron emits a message in the format:

[Vocab Size Padding] Padded vocab (size: 32761) with 39 dummy tokens (new size: 32800)

This confirms that 39 dummy tokens were added to reach the divisibility target.

Handling Tensor Parallelism and Custom Tokenizers

When using tensor parallelism with a world size greater than 1, the effective divisor becomes the product of your configured value and tp_world_size. For example, if you set make_vocab_size_divisible_by: 8 and train with tp_world_size = 4, the vocabulary must be divisible by 32.

If your custom tokenizer yields a non-standard vocabulary size such as 41000, Nanotron will pad it to 41024 (the next multiple of 32) by adding 24 dummy tokens. This ensures efficient sharding across tensor-parallel ranks without requiring you to manually resize embedding matrices or adjust the tokenizer.

Summary

• Configure the make_vocab_size_divisible_by field in ModelArgs to control divisibility constraints.
• Understand that the final divisor equals make_vocab_size_divisible_by × tp_world_size.
• Locate the padding logic in src/nanotron/helpers.py inside the _vocab_size_with_padding function.
• Verify padding through runtime logs or by inspecting model.config.vocab_size after trainer initialization.
• Apply settings via YAML, command-line arguments, or programmatic configuration objects.

Frequently Asked Questions

How do I disable vocabulary padding entirely?

Set make_vocab_size_divisible_by to 1 in your configuration. Since the default value is 1, Nanotron will only enforce divisibility by the tensor-parallel world size itself, adding no extra padding beyond what the TP layout requires.

Why is my final vocabulary size larger than my tokenizer's vocabulary?

Nanotron automatically pads the vocabulary to satisfy hardware alignment constraints. If you configure make_vocab_size_divisible_by: 8 and use a tokenizer with 31999 tokens, Nanotron adds one dummy token to reach 32000. These extra tokens are ignored during training and do not affect the model's semantic output.

Does this padding affect the output logits dimension?

Yes. The padded vocabulary size determines the final dimension of the language model head (output layer). While the padded tokens are technically part of the output space, they correspond to dummy embeddings that are never activated by valid input IDs, ensuring the model behavior remains identical to the unpadded version.

Where can I see how many dummy tokens were added?

Check the training logs for the warning message emitted by _vocab_size_with_padding in src/nanotron/helpers.py. The message explicitly states the original size, the number of dummy tokens added, and the new padded size, allowing you to verify the exact padding count applied by the trainer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}