Why the Llama3 Chinese Vocabulary Was Not Expanded: 4 Technical Reasons

The maintainers of crazyboym/llama3-chinese-chat deliberately retain the original 128k tokenizer to preserve Llama 3's pretrained knowledge, citing empirical evidence that vocabulary expansion degrades performance, while prioritizing high-quality Chinese instruction data instead.

The crazyboym/llama3-chinese-chat project adapts Meta’s Llama 3 for Chinese dialogue without modifying its underlying tokenizer vocabulary. While many Chinese LLM fine-tuning efforts inject thousands of Chinese-specific tokens into the embedding matrix, this repository keeps the original ~128,000 token set to safeguard the model’s multilingual capabilities and pretrained weights.

Preserving Pretrained Knowledge

Llama 3 was pretrained on 15 trillion tokens using its full 128k vocabulary vocabulary. Adding new Chinese tokens would break the alignment between the existing token embeddings and the language model’s learned weights. This misalignment risks eroding the broad multilingual capabilities baked into the original pretrained model, potentially degrading performance across all languages rather than just improving Chinese fluency.

Empirical Evidence From Llama2

The repository maintainers reference prior experiments showing that vocabulary expansion on earlier Llama 2 models "did not yield better performance". Based on this empirical data, they expect Llama 3 to suffer the same issue if subjected to similar modifications. The evidence suggests that simply increasing token count without corresponding high-quality pretraining data offers minimal gains while introducing instability.

Prioritizing Data Quality Over Vocabulary Size

Rather than altering the token set, the project roadmap emphasizes improving the quality of Chinese instruction data and fine-tuning techniques (SFT/DPO). The maintainers believe that a richer, well-curated Chinese dataset produces larger performance gains than expanding the vocabulary. This approach treats the tokenizer as a fixed constraint and focuses optimization efforts on post-training data curation and alignment methods.

The Expansion Script That Wasn't Used

Although the repository includes a utility to add new tokens, the README explicitly states this step is not performed for the Llama 3 Chinese models. The helper script tools/expand_embedding_and_lmhead.py copies the original embeddings for existing tokens and computes averaged embeddings for any new Chinese tokens, but remains unused in the default workflow.

As documented in the README.md#L25-L27, the Q&A section answers "词表扩充了吗？" (Was the vocabulary expanded?) with a clear negative, directing users to rely on the original tokenizer instead.

How to Use the Original Vocabulary

Since the repository preserves the original tokenizer, inference uses the standard Llama 3 token set without modification.

Example inference with the unchanged tokenizer:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "shareAI/llama3-Chinese-chat-8b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "<|begin_of_text|><<SYS>>\n你是一个中文助理。\n<</SYS>>\n\nUser: 你好！<|eot_id|>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(inputs.input_ids, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))

If you wanted to expand the vocabulary (not recommended by the authors):

python tools/expand_embedding_and_lmhead.py \
    --old_tokenizer ./original_llama3_tokenizer \
    --new_tokenizer ./chinese_tokenizer \
    --num_shards 1 \
    --old_model ./original_llama3_weights \
    --new_model ./expanded_llama3_weights \
    --save_embedding_plots

Summary

• Pretrained knowledge protection: The 128k vocabulary alignment from 15T tokens of pretraining would be disrupted by new tokens.
• Negative empirical results: Previous Llama 2 vocabulary expansions showed no performance benefit.
• Data-centric approach: High-quality Chinese SFT/DPO data provides better returns than tokenizer modification.
• Unused utilities: The tools/expand_embedding_and_lmhead.py script exists but is explicitly bypassed according to README.md.

Frequently Asked Questions

Did the Llama3 Chinese chat model expand its vocabulary?

No. According to the README.md Q&A section at lines 25-27, the repository explicitly did not expand the tokenizer vocabulary. The model uses the original Llama 3 tokenizer with approximately 128,000 tokens to preserve the integrity of the pretrained embeddings.

What is the risk of adding new tokens to a pretrained model?

Adding new tokens requires initializing new embedding vectors and LM head weights that were not present during the original 15-trillion-token pretraining. This breaks the alignment between the frozen pretrained weights and the new randomly initialized (or averaged) embeddings, potentially causing catastrophic forgetting or degraded multilingual performance across the entire model.

Is there a script to expand the vocabulary if I want to experiment?

Yes. The repository provides tools/expand_embedding_and_lmhead.py, which can add new tokens by copying original embeddings for existing tokens and computing averaged embeddings for new Chinese tokens. However, the maintainers do not use this script in their standard training pipeline and caution against it based on their Llama 2 experiments.

How does the model handle Chinese text without specialized tokens?

The original Llama 3 tokenizer already includes sufficient Unicode coverage to represent Chinese characters, albeit as multiple subword tokens rather than single dedicated tokens. The maintainers compensate for this by focusing on high-quality Chinese instruction data and supervised fine-tuning (SFT), teaching the model to generate fluent Chinese sequences using the existing multilingual token set rather than introducing new vocabulary items.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}