Evaluating Llama3 Chinese Models with the MMLU Benchmark: A Complete Guide

The crazyboym/llama3-chinese-chat repository reports MMLU scores ranging from 66.2% to 67.2% across four model variants, with instruction-tuned DPO versions achieving the highest 5-shot accuracy of 67.2% on the Massive Multitask Language Understanding benchmark.

The crazyboym/llama3-chinese-chat repository provides a complete pipeline for fine-tuning and evaluating Chinese-optimized Llama 3 models against standard academic benchmarks. This guide examines the reported MMLU evaluation results, explains the performance differences between base and instruction-tuned variants, and provides reproducible code examples for running your own assessments using the OpenCompass framework.

Understanding the Llama3 Chinese Model Variants

The repository maintains four primary checkpoints that are evaluated against the MMLU benchmark. Understanding these variants is essential for interpreting the performance metrics:

• LLaMA 3-8B (base): The original Llama 3 8-billion-parameter model with publicly released weights, serving as the baseline for comparison.
• LLaMA 3-8B (shareAI-V2): A variant incorporating additional Chinese pre-training data followed by a short supervised fine-tuning (SFT) stage to improve Chinese language understanding.
• LLaMA 3-8B Instruct: The instruction-tuned version refined with Direct Preference Optimization (DPO) to follow user instructions and adopt a chat-friendly style.
• LLaMA 3-8B Instruct (shareAI-V2): The combination of Chinese pre-training and DPO-based instruction tuning, representing the most optimized variant for Chinese conversational AI.

According to the source code analysis, these models are loaded via transformers.AutoModelForCausalLM as demonstrated in deploy/python/chat_demo.py, with each variant exposing the same generation API but differing in their underlying weight configurations.

MMLU Benchmark Results and Performance Analysis

The repository reports standardized 5-shot MMLU results alongside Chinese-specific evaluations. The following scores are documented in the README at lines 505-516:

Model	MMLU (5-shot)	C-Eval (5-shot)	TriviaQA-Wiki (8-shot)
LLaMA 3-8B	66.6	49.8	81.4
LLaMA 3-8B (shareAI-V2)	66.2	50.9	81.8
LLaMA 3-8B Instruct	67.1	—	—
LLaMA 3-8B Instruct (shareAI-V2)	67.2	—	—

MMLU measures broad knowledge across 57 subjects spanning science, humanities, and social sciences. For an 8-billion-parameter model, scores in the mid-60s represent strong general reasoning capabilities. The data reveals that DPO-based instruction tuning consistently improves MMLU performance, with the shareAI-V2 instruct variant achieving the highest score of 67.2%.

While the shareAI-V2 base model shows a slight MMLU decrease (66.2% vs. 66.6%), it achieves superior performance on C-Eval, a Chinese-specific benchmark covering 52 tasks including reading comprehension and code generation. This trade-off indicates that additional Chinese pre-training enhances domestic domain knowledge while maintaining competitive general reasoning scores.

Reproducing the MMLU Evaluation

The repository utilizes the OpenCompass evaluation framework to generate reproducible benchmark scores. You can replicate the reported results using the following workflow.

Loading Models for Inference

The evaluation pipeline begins with loading the checkpoint via the Hugging Face transformers library. The following Python code mirrors the implementation in deploy/python/chat_demo.py:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Select the instruct-DPO checkpoint for best MMLU performance

model_name = "shareAI/llama3-Chinese-instruct-DPO-beta0.5"

tokenizer = AutoTokenizer.from_pretrained(
    model_name, 
    trust_remote_code=True, 
    use_fast=False
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
    low_cpu_mem_usage=True,
)

def generate_response(query, system="You are a helpful Chinese assistant."):
    """Generate response using the Llama3 Chinese chat template."""
    prompt = f"<|begin_of_text|><<SYS>>\n{system}\n<</SYS>>\n\n{query}<|eot_id|>"
    input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
    
    output_ids = model.generate(
        input_ids,
        max_new_tokens=256,
        do_sample=True,
        top_p=0.9,
        temperature=0.6,
        repetition_penalty=1.1,
        eos_token_id=tokenizer.encode("<|end_of_text|>")[0],
    )
    
    response = tokenizer.decode(
        output_ids[0][input_ids.shape[-1]:], 
        skip_special_tokens=True
    )
    return response.strip()

# Example usage for manual verification

print(generate_response("请解释一下“举一反三”这个成语的含义。"))

This inference logic is utilized by OpenCompass when feeding 5-shot prompts to the model during benchmark execution.

Configuring the OpenCompass Benchmark Harness

To run the official MMLU evaluation matching the repository's reported scores, install OpenCompass and create a configuration file pointing to your local checkpoint:

# Install the specific OpenCompass version used in the repository

pip install "opencompass==0.2.0"

Create a configuration file named mmlu_llama3_config.py:

from mmengine import Config
from opencompass.datasets import MMLUDataset
from opencompass.models import HuggingFaceCausalLM

# Model configuration matching the repository's setup

model = dict(
    type=HuggingFaceCausalLM,
    path="shareAI/llama3-Chinese-instruct-DPO-beta0.5",
    tokenizer="shareAI/llama3-Chinese-instruct-DPO-beta0.5",
    max_seq_len=4096,
    batch_size=8,
    generation_kwargs=dict(
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    ),
)

# MMLU dataset with 5-shot prompting

datasets = [
    dict(
        type=MMLUDataset,
        name="mmlu",
        split="test",
        path="opencompass/mmlu",
        few_shot=5,
    ),
]

# Evaluation metric configuration

evaluation = dict(
    eval_type="accuracy",
    metric="accuracy",
)

# Complete configuration assembly

_cfg = dict(
    model=model,
    datasets=datasets,
    evaluation=evaluation,
    work_dir="./work_dirs/mmlu_llama3_instruct",
)

return Config(_cfg)

Execute the benchmark with:

python -m opencompass run mmlu_llama3_config.py

The harness will process the 57 MMLU subjects using 5-shot prompting and output an aggregate accuracy score. According to the repository documentation, this process should yield approximately 67.2% accuracy for the instruct-DPO variant when run on GPU-enabled hardware.

Key Source Files for Evaluation

The following files in crazyboym/llama3-chinese-chat provide the complete implementation details for the MMLU evaluation pipeline:

• deploy/python/chat_demo.py: Core inference script demonstrating model loading, prompt templating, and generation parameters used during evaluation.
• deploy/API/easy_server_demo.py: Flask-based API server implementation that OpenCompass can query when running benchmarks in server mode.
• tools/convert_raw_data_for_firefly.py: Data preprocessing utility that converts raw Chinese corpora into ShareGPT format, enabling the training improvements that boost C-Eval and MMLU scores.
• README.md (lines 505-516): Contains the official evaluation table with MMLU, C-Eval, and TriviaQA-Wiki results, along with links to the OpenCompass configuration citations.

These files collectively demonstrate how the models are built, served, and evaluated against the MMLU benchmark.

Summary

• The crazyboym/llama3-chinese-chat repository achieves MMLU scores between 66.2% and 67.2% across its four Llama 3 8B variants, with instruction-tuned DPO models performing best.
• Direct Preference Optimization (DPO) consistently improves MMLU performance, adding approximately 0.5-0.6 percentage points over base models.
• The shareAI-V2 pre-training variant optimizes for Chinese-specific benchmarks (C-Eval) while maintaining competitive general reasoning scores on MMLU.
• Reproducible evaluation is enabled through OpenCompass integration, with configuration examples available in the repository's documentation.
• Key implementation files including deploy/python/chat_demo.py provide the exact inference logic used to generate the reported benchmark scores.

Frequently Asked Questions

What is the MMLU benchmark and why is it important for Chinese LLMs?

MMLU (Massive Multitask Language Understanding) is a standardized benchmark testing knowledge across 57 subjects including mathematics, history, law, and medicine. For Chinese LLMs, strong MMLU scores indicate that the model retains broad world knowledge despite being fine-tuned on Chinese corpora, ensuring the model is useful for both general reasoning and Chinese-specific tasks.

How does the shareAI-V2 pre-training affect MMLU scores compared to the base model?

The shareAI-V2 variant shows a marginal MMLU decrease from 66.6% to 66.2% compared to the base model, but achieves superior performance on C-Eval (50.9% vs. 49.8%). This indicates that additional Chinese pre-training slightly shifts the model's knowledge distribution toward Chinese domains without significantly compromising general English academic knowledge.

Can I run the MMLU evaluation without OpenCompass?

While the repository officially uses OpenCompass for standardized reporting, you can implement custom MMLU evaluation using the lm-evaluation-harness or a custom Python script utilizing the generate_response function from deploy/python/chat_demo.py. However, OpenCompass is recommended for exact score reproduction due to its standardized 5-shot prompting and aggregation methodologies.

What hardware is required to reproduce the 67.2% MMLU score?

Reproducing the reported scores requires a GPU with sufficient VRAM to load the 8-billion-parameter model in FP16 precision (approximately 16GB VRAM minimum). The OpenCompass evaluation can be distributed across multiple GPUs using the device_map="auto" configuration shown in the loading examples, or run sequentially on a single high-memory GPU.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}