# Evaluating Llama3 Chinese Models with the MMLU Benchmark: A Complete Guide

> Discover Llama3 Chinese model performance on the MMLU benchmark. See how instruction-tuned DPO versions achieve top 5-shot accuracy and explore the results from the crazyboym/llama3-chinese-chat repository.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: testing
- Published: 2026-02-28

---

**The crazyboym/llama3-chinese-chat repository reports MMLU scores ranging from 66.2% to 67.2% across four model variants, with instruction-tuned DPO versions achieving the highest 5-shot accuracy of 67.2% on the Massive Multitask Language Understanding benchmark.**

The **crazyboym/llama3-chinese-chat** repository provides a complete pipeline for fine-tuning and evaluating Chinese-optimized Llama 3 models against standard academic benchmarks. This guide examines the reported MMLU evaluation results, explains the performance differences between base and instruction-tuned variants, and provides reproducible code examples for running your own assessments using the OpenCompass framework.

## Understanding the Llama3 Chinese Model Variants

The repository maintains four primary checkpoints that are evaluated against the MMLU benchmark. Understanding these variants is essential for interpreting the performance metrics:

- **LLaMA 3-8B (base)**: The original Llama 3 8-billion-parameter model with publicly released weights, serving as the baseline for comparison.
- **LLaMA 3-8B (shareAI-V2)**: A variant incorporating additional Chinese pre-training data followed by a short supervised fine-tuning (SFT) stage to improve Chinese language understanding.
- **LLaMA 3-8B Instruct**: The instruction-tuned version refined with Direct Preference Optimization (DPO) to follow user instructions and adopt a chat-friendly style.
- **LLaMA 3-8B Instruct (shareAI-V2)**: The combination of Chinese pre-training and DPO-based instruction tuning, representing the most optimized variant for Chinese conversational AI.

According to the source code analysis, these models are loaded via `transformers.AutoModelForCausalLM` as demonstrated in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py), with each variant exposing the same generation API but differing in their underlying weight configurations.

## MMLU Benchmark Results and Performance Analysis

The repository reports standardized 5-shot MMLU results alongside Chinese-specific evaluations. The following scores are documented in the README at lines 505-516:

| Model | MMLU (5-shot) | C-Eval (5-shot) | TriviaQA-Wiki (8-shot) |
|-------|---------------|-----------------|------------------------|
| LLaMA 3-8B | **66.6** | 49.8 | 81.4 |
| LLaMA 3-8B (shareAI-V2) | 66.2 | **50.9** | **81.8** |
| LLaMA 3-8B Instruct | **67.1** | — | — |
| LLaMA 3-8B Instruct (shareAI-V2) | **67.2** | — | — |

**MMLU** measures broad knowledge across 57 subjects spanning science, humanities, and social sciences. For an 8-billion-parameter model, scores in the mid-60s represent strong general reasoning capabilities. The data reveals that **DPO-based instruction tuning consistently improves MMLU performance**, with the shareAI-V2 instruct variant achieving the highest score of 67.2%.

While the shareAI-V2 base model shows a slight MMLU decrease (66.2% vs. 66.6%), it achieves superior performance on **C-Eval**, a Chinese-specific benchmark covering 52 tasks including reading comprehension and code generation. This trade-off indicates that additional Chinese pre-training enhances domestic domain knowledge while maintaining competitive general reasoning scores.

## Reproducing the MMLU Evaluation

The repository utilizes the **OpenCompass** evaluation framework to generate reproducible benchmark scores. You can replicate the reported results using the following workflow.

### Loading Models for Inference

The evaluation pipeline begins with loading the checkpoint via the Hugging Face `transformers` library. The following Python code mirrors the implementation in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py):

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Select the instruct-DPO checkpoint for best MMLU performance

model_name = "shareAI/llama3-Chinese-instruct-DPO-beta0.5"

tokenizer = AutoTokenizer.from_pretrained(
    model_name, 
    trust_remote_code=True, 
    use_fast=False
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True,
    low_cpu_mem_usage=True,
)

def generate_response(query, system="You are a helpful Chinese assistant."):
    """Generate response using the Llama3 Chinese chat template."""
    prompt = f"<|begin_of_text|><<SYS>>\n{system}\n<</SYS>>\n\n{query}<|eot_id|>"
    input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)
    
    output_ids = model.generate(
        input_ids,
        max_new_tokens=256,
        do_sample=True,
        top_p=0.9,
        temperature=0.6,
        repetition_penalty=1.1,
        eos_token_id=tokenizer.encode("<|end_of_text|>")[0],
    )
    
    response = tokenizer.decode(
        output_ids[0][input_ids.shape[-1]:], 
        skip_special_tokens=True
    )
    return response.strip()

# Example usage for manual verification

print(generate_response("请解释一下“举一反三”这个成语的含义。"))

```

This inference logic is utilized by OpenCompass when feeding 5-shot prompts to the model during benchmark execution.

### Configuring the OpenCompass Benchmark Harness

To run the official MMLU evaluation matching the repository's reported scores, install OpenCompass and create a configuration file pointing to your local checkpoint:

```bash

# Install the specific OpenCompass version used in the repository

pip install "opencompass==0.2.0"

```

Create a configuration file named [`mmlu_llama3_config.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/mmlu_llama3_config.py):

```python
from mmengine import Config
from opencompass.datasets import MMLUDataset
from opencompass.models import HuggingFaceCausalLM

# Model configuration matching the repository's setup

model = dict(
    type=HuggingFaceCausalLM,
    path="shareAI/llama3-Chinese-instruct-DPO-beta0.5",
    tokenizer="shareAI/llama3-Chinese-instruct-DPO-beta0.5",
    max_seq_len=4096,
    batch_size=8,
    generation_kwargs=dict(
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
    ),
)

# MMLU dataset with 5-shot prompting

datasets = [
    dict(
        type=MMLUDataset,
        name="mmlu",
        split="test",
        path="opencompass/mmlu",
        few_shot=5,
    ),
]

# Evaluation metric configuration

evaluation = dict(
    eval_type="accuracy",
    metric="accuracy",
)

# Complete configuration assembly

_cfg = dict(
    model=model,
    datasets=datasets,
    evaluation=evaluation,
    work_dir="./work_dirs/mmlu_llama3_instruct",
)

return Config(_cfg)

```

Execute the benchmark with:

```bash
python -m opencompass run mmlu_llama3_config.py

```

The harness will process the 57 MMLU subjects using 5-shot prompting and output an aggregate accuracy score. According to the repository documentation, this process should yield approximately **67.2%** accuracy for the instruct-DPO variant when run on GPU-enabled hardware.

## Key Source Files for Evaluation

The following files in `crazyboym/llama3-chinese-chat` provide the complete implementation details for the MMLU evaluation pipeline:

- **[`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)**: Core inference script demonstrating model loading, prompt templating, and generation parameters used during evaluation.
- **[`deploy/API/easy_server_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/API/easy_server_demo.py)**: Flask-based API server implementation that OpenCompass can query when running benchmarks in server mode.
- **[`tools/convert_raw_data_for_firefly.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/tools/convert_raw_data_for_firefly.py)**: Data preprocessing utility that converts raw Chinese corpora into ShareGPT format, enabling the training improvements that boost C-Eval and MMLU scores.
- **[`README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/README.md)** (lines 505-516): Contains the official evaluation table with MMLU, C-Eval, and TriviaQA-Wiki results, along with links to the OpenCompass configuration citations.

These files collectively demonstrate how the models are built, served, and evaluated against the MMLU benchmark.

## Summary

- The **crazyboym/llama3-chinese-chat** repository achieves **MMLU scores between 66.2% and 67.2%** across its four Llama 3 8B variants, with instruction-tuned DPO models performing best.
- **Direct Preference Optimization (DPO)** consistently improves MMLU performance, adding approximately 0.5-0.6 percentage points over base models.
- The **shareAI-V2** pre-training variant optimizes for Chinese-specific benchmarks (C-Eval) while maintaining competitive general reasoning scores on MMLU.
- Reproducible evaluation is enabled through **OpenCompass** integration, with configuration examples available in the repository's documentation.
- Key implementation files including [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py) provide the exact inference logic used to generate the reported benchmark scores.

## Frequently Asked Questions

### What is the MMLU benchmark and why is it important for Chinese LLMs?

**MMLU (Massive Multitask Language Understanding)** is a standardized benchmark testing knowledge across 57 subjects including mathematics, history, law, and medicine. For Chinese LLMs, strong MMLU scores indicate that the model retains broad world knowledge despite being fine-tuned on Chinese corpora, ensuring the model is useful for both general reasoning and Chinese-specific tasks.

### How does the shareAI-V2 pre-training affect MMLU scores compared to the base model?

The shareAI-V2 variant shows a marginal MMLU decrease from 66.6% to 66.2% compared to the base model, but achieves superior performance on C-Eval (50.9% vs. 49.8%). This indicates that additional Chinese pre-training slightly shifts the model's knowledge distribution toward Chinese domains without significantly compromising general English academic knowledge.

### Can I run the MMLU evaluation without OpenCompass?

While the repository officially uses **OpenCompass** for standardized reporting, you can implement custom MMLU evaluation using the `lm-evaluation-harness` or a custom Python script utilizing the `generate_response` function from [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py). However, OpenCompass is recommended for exact score reproduction due to its standardized 5-shot prompting and aggregation methodologies.

### What hardware is required to reproduce the 67.2% MMLU score?

Reproducing the reported scores requires a GPU with sufficient VRAM to load the 8-billion-parameter model in FP16 precision (approximately 16GB VRAM minimum). The OpenCompass evaluation can be distributed across multiple GPUs using the `device_map="auto"` configuration shown in the loading examples, or run sequentially on a single high-memory GPU.