# Evaluating Llama3 Chinese Models with the C-Eval Benchmark: A Complete Guide

> Evaluate Llama3 Chinese models with the C-Eval benchmark using the complete inference pipeline from crazyboym/llama3-chinese-chat repository for reproducible results.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: evaluating-models
- Published: 2026-02-28

---

**The `crazyboym/llama3-chinese-chat` repository provides a complete inference pipeline and reports C-Eval scores of 49.8 for the base Llama 3-8B and 50.9 for the shareAI-V2 fine-tuned variant, with reproducible evaluation code available in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py).**

Evaluating Chinese language capabilities of large language models requires specialized benchmarks like C-Eval (Chinese-Eval), a comprehensive assessment suite covering 52 disciplines from Chinese education. The `llama3-chinese-chat` repository by crazyboym implements a full-stack solution for adapting Meta's Llama 3 to Chinese contexts, including the training scripts, model loaders, and prompt templates necessary to reproduce published C-Eval metrics.

## Understanding the C-Eval Benchmark Architecture

C-Eval tests knowledge and reasoning across humanities, social sciences, STEM, and professional domains using multiple-choice questions. The **5-shot evaluation** protocol—where five examples precede the test question—standardizes comparisons between base and fine-tuned models. According to the repository's README, this methodology produced the reported scores showing modest gains from the shareAI fine-tuning process.

## Repository Components for Model Evaluation

The evaluation pipeline rests on three tightly integrated components defined in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py).

### Model and Tokenizer Loading

The `load_model` function initializes the backbone using Hugging Face Transformers with specific optimizations for inference:

```python
from deploy.python.chat_demo import load_model, load_tokenizer

model = load_model(
    model_name_or_path="shareAI/llama3-Chinese-chat-8b",
    load_in_4bit=False,  # Set True for int4 quantization (~8GB VRAM)

    torch_dtype=torch.float16,
    device_map='auto'
)
tokenizer = load_tokenizer("shareAI/llama3-Chinese-chat-8b")

```

This loader supports both full precision and **4-bit quantized** checkpoints via `load_in_4bit=True`, enabling evaluation on consumer hardware. The function automatically handles LoRA/PEFT adapter merging when present.

### Prompt Template System

Evaluation consistency depends on rigid prompt formatting. The repository defines a `Template` dataclass that encapsulates the Llama 3 chat format:

```python
from deploy.python.chat_demo import template_dict, build_prompt

template = template_dict["llama3"]
stop_id = tokenizer.encode(template.stop_word, add_special_tokens=True)[0]

```

The `build_prompt` function assembles system prompts, conversation history, and current queries into tokenized tensors. For C-Eval testing, you invoke it with empty history to isolate per-question performance:

```python
input_ids = build_prompt(
    tokenizer, 
    template, 
    question_text, 
    history=[],  # No prior context for benchmark consistency

    system=None
).to(model.device)

```

### Inference Loop and Generation Parameters

The generation logic uses `model.generate` with deterministic settings suitable for standardized testing:

```python
output_ids = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=False,  # Greedy decoding for reproducibility

    eos_token_id=stop_id,
    repetition_penalty=1.0
)[0][len(input_ids[0]):]

prediction = tokenizer.decode(output_ids).strip().replace(template.stop_word, "").strip()

```

The `stop_id` derives from the template's `stop_word` field (typically `<|end_of_text|>`), ensuring generation terminates correctly without polluting the answer extraction.

## Reported C-Eval Performance Metrics

The README documentation provides verified 5-shot C-Eval results comparing base and adapted models:

| Model Variant | 5-Shot C-Eval Score |
|--------------|---------------------|
| **LLaMA 3-8B (Base)** | 49.8 |
| **LLaMA 3-8B (shareAI-V2)** | 50.9 |

The **1.1 percentage point improvement** demonstrates that the shareAI fine-tuning process—documented in [`train/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/train/README.md)—enhances Chinese knowledge retention without catastrophic forgetting of English capabilities.

## Reproducing the C-Eval Benchmark

To replicate these results independently, implement the following evaluation harness using the repository's utilities.

### Environment Preparation

1. Install pinned dependencies as specified in the training documentation:
   ```bash
   pip install transformers==4.40.1 peft torch
   ```

2. Download the C-Eval dataset from the official repository and preprocess it into (question, answer) pairs.

### Evaluation Implementation

The following script implements exact-match scoring against C-Eval's multiple-choice format:

```python
import torch
from deploy.python.chat_demo import load_model, load_tokenizer, build_prompt, template_dict
from transformers import AutoTokenizer

def evaluate_ceval_item(model, tokenizer, template, question: str, choices: list, answer: str) -> bool:
    """
    Evaluate a single C-Eval multiple choice item.
    Assumes answer is one of: A, B, C, D
    """
    # Format question with choices

    prompt = f"{question}\n" + "\n".join([f"{chr(65+i)}. {c}" for i, c in enumerate(choices)])
    prompt += "\n答案："
    
    # Build input

    input_ids = build_prompt(tokenizer, template, prompt, [], system=None).to(model.device)
    
    # Generate

    stop_id = tokenizer.encode(template.stop_word, add_special_tokens=True)[0]
    output_ids = model.generate(
        input_ids,
        max_new_tokens=10,  # Short generation for single letter

        do_sample=False,
        eos_token_id=stop_id,
    )[0][len(input_ids[0]):]
    
    pred = tokenizer.decode(output_ids).strip().replace(template.stop_word, "").strip()
    
    # Extract first character and compare

    return pred[0].upper() == answer.upper()

# Batch evaluation loop

model = load_model("shareAI/llama3-Chinese-chat-8b", load_in_4bit=True)
tokenizer = load_tokenizer("shareAI/llama3-Chinese-chat-8b")
template = template_dict["llama3"]

correct = 0
total = 0

# Iterate your C-Eval dataset here

for item in ceval_dataset:
    if evaluate_ceval_item(model, tokenizer, template, item["question"], item["choices"], item["answer"]):
        correct += 1
    total += 1

print(f"C-Eval Accuracy: {correct/total*100:.1f}%")

```

## Deployment Options for Evaluation at Scale

Beyond the Python CLI demo, the repository supports production-grade evaluation infrastructure.

**Interactive CLI Demo**
Run the reference implementation directly:

```bash
python deploy/python/chat_demo.py

```

**Streamlit Web Interface**
For human-in-the-loop validation of C-Eval samples:

```bash
pip install streamlit
streamlit run deploy/streamlit/web_llama3_chat.py /path/to/checkpoint

```

**vLLM OpenAI-Compatible Server**
For high-throughput batch evaluation:

```bash
cd deploy/vLLM
python -m vllm.entrypoints.openai.api_server \
    --model shareAI/llama3-Chinese-chat-8b \
    --dtype half \
    --max-model-len 4096

```

Query the endpoint with standard OpenAI SDK patterns to programmatically submit C-Eval questions and parse JSON responses for automated scoring.

## Summary

- The `crazyboym/llama3-chinese-chat` repository implements a complete evaluation stack in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py), featuring `load_model`, `load_tokenizer`, and `build_prompt` functions.
- Reported 5-shot C-Eval scores show the shareAI-V2 fine-tuned model (50.9) outperforming the base Llama 3-8B checkpoint (49.8) by 1.1 points.
- The `Template` dataclass ensures consistent prompt formatting required for reproducible benchmark results.
- Support for 4-bit quantization via `load_in_4bit=True` enables C-Eval testing on single consumer GPUs with approximately 8GB VRAM.
- Multiple deployment paths—CLI, Streamlit, and vLLM—accommodate both individual researcher validation and large-scale automated evaluation.

## Frequently Asked Questions

### What is C-Eval and why is it critical for Chinese Llama3 models?

C-Eval (Chinese-Eval) is a multi-disciplinary benchmark covering 52 subjects from the Chinese education system, testing advanced knowledge and reasoning. Unlike English-centric benchmarks such as MMLU, C-Eval specifically measures how well models like Llama 3 handle Chinese cultural context, idiomatic expressions, and domain-specific terminology after fine-tuning.

### How does the shareAI-V2 model achieve higher C-Eval scores than the base model?

The shareAI-V2 variant, documented in [`train/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/train/README.md), employs supervised fine-tuning (SFT) on curated Chinese instruction data and potentially DPO (Direct Preference Optimization). This adaptation improves alignment with Chinese answer patterns and knowledge retrieval, yielding the 50.9 score compared to the base model's 49.8 on 5-shot C-Eval.

### Can I evaluate these models on C-Eval without a high-end GPU?

Yes. The `load_model` function in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py) accepts `load_in_4bit=True`, which loads the 8B parameter model in approximately 8GB of VRAM using bitsandbytes quantization. This enables C-Eval evaluation on consumer GPUs like the RTX 3090 or RTX 4090, though generation speed will be slower than full-precision inference.

### Where is the prompt template defined for standardized C-Eval testing?

The prompt template is defined in the `Template` dataclass within [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py) (lines 8-40). The `template_dict["llama3"]` entry contains the system prompt format, user/assistant delimiters, and stop word (`<|end_of_text|>`) required to match the official Llama 3 chat format used during the reported C-Eval testing.