Evaluating Llama3 Chinese Models with the C-Eval Benchmark: A Complete Guide

The crazyboym/llama3-chinese-chat repository provides a complete inference pipeline and reports C-Eval scores of 49.8 for the base Llama 3-8B and 50.9 for the shareAI-V2 fine-tuned variant, with reproducible evaluation code available in deploy/python/chat_demo.py.

Evaluating Chinese language capabilities of large language models requires specialized benchmarks like C-Eval (Chinese-Eval), a comprehensive assessment suite covering 52 disciplines from Chinese education. The llama3-chinese-chat repository by crazyboym implements a full-stack solution for adapting Meta's Llama 3 to Chinese contexts, including the training scripts, model loaders, and prompt templates necessary to reproduce published C-Eval metrics.

Understanding the C-Eval Benchmark Architecture

C-Eval tests knowledge and reasoning across humanities, social sciences, STEM, and professional domains using multiple-choice questions. The 5-shot evaluation protocol—where five examples precede the test question—standardizes comparisons between base and fine-tuned models. According to the repository's README, this methodology produced the reported scores showing modest gains from the shareAI fine-tuning process.

Repository Components for Model Evaluation

The evaluation pipeline rests on three tightly integrated components defined in deploy/python/chat_demo.py.

Model and Tokenizer Loading

The load_model function initializes the backbone using Hugging Face Transformers with specific optimizations for inference:

from deploy.python.chat_demo import load_model, load_tokenizer

model = load_model(
    model_name_or_path="shareAI/llama3-Chinese-chat-8b",
    load_in_4bit=False,  # Set True for int4 quantization (~8GB VRAM)

    torch_dtype=torch.float16,
    device_map='auto'
)
tokenizer = load_tokenizer("shareAI/llama3-Chinese-chat-8b")

This loader supports both full precision and 4-bit quantized checkpoints via load_in_4bit=True, enabling evaluation on consumer hardware. The function automatically handles LoRA/PEFT adapter merging when present.

Prompt Template System

Evaluation consistency depends on rigid prompt formatting. The repository defines a Template dataclass that encapsulates the Llama 3 chat format:

from deploy.python.chat_demo import template_dict, build_prompt

template = template_dict["llama3"]
stop_id = tokenizer.encode(template.stop_word, add_special_tokens=True)[0]

The build_prompt function assembles system prompts, conversation history, and current queries into tokenized tensors. For C-Eval testing, you invoke it with empty history to isolate per-question performance:

input_ids = build_prompt(
    tokenizer, 
    template, 
    question_text, 
    history=[],  # No prior context for benchmark consistency

    system=None
).to(model.device)

Inference Loop and Generation Parameters

The generation logic uses model.generate with deterministic settings suitable for standardized testing:

output_ids = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=False,  # Greedy decoding for reproducibility

    eos_token_id=stop_id,
    repetition_penalty=1.0
)[0][len(input_ids[0]):]

prediction = tokenizer.decode(output_ids).strip().replace(template.stop_word, "").strip()

The stop_id derives from the template's stop_word field (typically <|end_of_text|>), ensuring generation terminates correctly without polluting the answer extraction.

Reported C-Eval Performance Metrics

The README documentation provides verified 5-shot C-Eval results comparing base and adapted models:

Model Variant	5-Shot C-Eval Score
LLaMA 3-8B (Base)	49.8
LLaMA 3-8B (shareAI-V2)	50.9

The 1.1 percentage point improvement demonstrates that the shareAI fine-tuning process—documented in train/README.md—enhances Chinese knowledge retention without catastrophic forgetting of English capabilities.

Reproducing the C-Eval Benchmark

To replicate these results independently, implement the following evaluation harness using the repository's utilities.

Environment Preparation

1. Install pinned dependencies as specified in the training documentation:

   pip install transformers==4.40.1 peft torch

2. Download the C-Eval dataset from the official repository and preprocess it into (question, answer) pairs.

Evaluation Implementation

The following script implements exact-match scoring against C-Eval's multiple-choice format:

import torch
from deploy.python.chat_demo import load_model, load_tokenizer, build_prompt, template_dict
from transformers import AutoTokenizer

def evaluate_ceval_item(model, tokenizer, template, question: str, choices: list, answer: str) -> bool:
    """
    Evaluate a single C-Eval multiple choice item.
    Assumes answer is one of: A, B, C, D
    """
    # Format question with choices

    prompt = f"{question}\n" + "\n".join([f"{chr(65+i)}. {c}" for i, c in enumerate(choices)])
    prompt += "\n答案："
    
    # Build input

    input_ids = build_prompt(tokenizer, template, prompt, [], system=None).to(model.device)
    
    # Generate

    stop_id = tokenizer.encode(template.stop_word, add_special_tokens=True)[0]
    output_ids = model.generate(
        input_ids,
        max_new_tokens=10,  # Short generation for single letter

        do_sample=False,
        eos_token_id=stop_id,
    )[0][len(input_ids[0]):]
    
    pred = tokenizer.decode(output_ids).strip().replace(template.stop_word, "").strip()
    
    # Extract first character and compare

    return pred[0].upper() == answer.upper()

# Batch evaluation loop

model = load_model("shareAI/llama3-Chinese-chat-8b", load_in_4bit=True)
tokenizer = load_tokenizer("shareAI/llama3-Chinese-chat-8b")
template = template_dict["llama3"]

correct = 0
total = 0

# Iterate your C-Eval dataset here

for item in ceval_dataset:
    if evaluate_ceval_item(model, tokenizer, template, item["question"], item["choices"], item["answer"]):
        correct += 1
    total += 1

print(f"C-Eval Accuracy: {correct/total*100:.1f}%")

Deployment Options for Evaluation at Scale

Beyond the Python CLI demo, the repository supports production-grade evaluation infrastructure.

Interactive CLI Demo Run the reference implementation directly:

python deploy/python/chat_demo.py

Streamlit Web Interface For human-in-the-loop validation of C-Eval samples:

pip install streamlit
streamlit run deploy/streamlit/web_llama3_chat.py /path/to/checkpoint

vLLM OpenAI-Compatible Server For high-throughput batch evaluation:

cd deploy/vLLM
python -m vllm.entrypoints.openai.api_server \
    --model shareAI/llama3-Chinese-chat-8b \
    --dtype half \
    --max-model-len 4096

Query the endpoint with standard OpenAI SDK patterns to programmatically submit C-Eval questions and parse JSON responses for automated scoring.

Summary

• The crazyboym/llama3-chinese-chat repository implements a complete evaluation stack in deploy/python/chat_demo.py, featuring load_model, load_tokenizer, and build_prompt functions.
• Reported 5-shot C-Eval scores show the shareAI-V2 fine-tuned model (50.9) outperforming the base Llama 3-8B checkpoint (49.8) by 1.1 points.
• The Template dataclass ensures consistent prompt formatting required for reproducible benchmark results.
• Support for 4-bit quantization via load_in_4bit=True enables C-Eval testing on single consumer GPUs with approximately 8GB VRAM.
• Multiple deployment paths—CLI, Streamlit, and vLLM—accommodate both individual researcher validation and large-scale automated evaluation.

Frequently Asked Questions

What is C-Eval and why is it critical for Chinese Llama3 models?

C-Eval (Chinese-Eval) is a multi-disciplinary benchmark covering 52 subjects from the Chinese education system, testing advanced knowledge and reasoning. Unlike English-centric benchmarks such as MMLU, C-Eval specifically measures how well models like Llama 3 handle Chinese cultural context, idiomatic expressions, and domain-specific terminology after fine-tuning.

How does the shareAI-V2 model achieve higher C-Eval scores than the base model?

The shareAI-V2 variant, documented in train/README.md, employs supervised fine-tuning (SFT) on curated Chinese instruction data and potentially DPO (Direct Preference Optimization). This adaptation improves alignment with Chinese answer patterns and knowledge retrieval, yielding the 50.9 score compared to the base model's 49.8 on 5-shot C-Eval.

Can I evaluate these models on C-Eval without a high-end GPU?

Yes. The load_model function in deploy/python/chat_demo.py accepts load_in_4bit=True, which loads the 8B parameter model in approximately 8GB of VRAM using bitsandbytes quantization. This enables C-Eval evaluation on consumer GPUs like the RTX 3090 or RTX 4090, though generation speed will be slower than full-precision inference.

Where is the prompt template defined for standardized C-Eval testing?

The prompt template is defined in the Template dataclass within deploy/python/chat_demo.py (lines 8-40). The template_dict["llama3"] entry contains the system prompt format, user/assistant delimiters, and stop word (<|end_of_text|>) required to match the official Llama 3 chat format used during the reported C-Eval testing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}