LLM and Agent Evaluation Frameworks and Benchmarks: A Complete Guide from ai-engineering-from-scratch

The curriculum teaches a layered evaluation strategy spanning from classic academic benchmarks like MMLU-Pro and HumanEval to production-grade harnesses, calibration metrics, and safety-specific benchmarks for assessing both base models and agent systems.

This guide distills the complete evaluation methodology taught in the rohitg00/ai-engineering-from-scratch repository. Whether you are benchmarking foundation models or production RAG pipelines, these evaluation frameworks provide the quantitative rigor needed to move from prototype to deployment.

Core Academic Benchmarks for LLM Assessment

The foundational lessons introduce canonical test collections that serve as the de-facto standard for assessing language model capabilities. These benchmarks appear in phases/10-llms-from-scratch/10-evaluation/docs/en.md.

MMLU-Pro (Massive Multitask Language Understanding) evaluates knowledge across approximately 15,000 multiple-choice questions spanning 57 subjects. This benchmark serves as the primary "knowledge" metric for general intelligence assessment.

HumanEval tests functional correctness through 164 Python programming problems, measuring a model's ability to generate executable, logically sound code.

MT-Bench-v2 provides a chat-oriented evaluation framework that scores helpfulness, factuality, and safety in conversational contexts. This appears in the fine-tuning pipeline documentation at phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md.

Additional task-specific suites include SWE-Bench for software engineering, MATH and GSM8K for arithmetic reasoning, and ARC with HellaSwag for commonsense inference.

The lm-evaluation-harness Framework

Moving beyond individual benchmarks, the curriculum teaches EleutherAI's lm-evaluation-harness, a community-maintained framework aggregating over 200 tasks. This harness is introduced in phases/11-llm-engineering/10-evaluation/docs/en.md.

The harness provides three critical capabilities:

• Uniform command-line interface: Execute evaluations via lm_eval --model <path> --tasks <list>
• Automatic metric aggregation: Computes accuracy, F1, exact-match, BLEU, and ROUGE scores consistently across tasks
• Plugin support: Extends to custom evaluations like RAGAS for retrieval-augmented generation

# Running MMLU and HumanEval via the harness

from lm_eval import LMEval

model = "EleutherAI/gpt-neo-2.7B"
tasks = ["mmlu", "humaneval"]
results = LMEval(model=model, tasks=tasks).run()
print(results)  # {'mmlu': 68.4, 'humaneval': 71.2}

Calibration and Perplexity Metrics

Raw accuracy scores prove insufficient for production deployments. The curriculum teaches Expected Calibration Error (ECE) and perplexity as complementary diagnostics in phases/19-capstone-projects/73-perplexity-calibration/docs/en.md.

ECE quantifies how well predicted probabilities align with empirical correctness. A well-calibrated model with 0.9 confidence should be correct 90% of the time.

Perplexity measures model confidence on held-out text corpora, offering insight into generative capabilities distinct from downstream task performance.

import numpy as np

def ece(confidences, labels, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    bin_idxs = np.digitize(confidences, bins) - 1
    ece_score = 0.0
    
    for b in range(n_bins):
        mask = bin_idxs == b
        if mask.sum() == 0:
            continue
        acc = labels[mask].mean()
        avg_conf = confidences[mask].mean()
        ece_score += np.abs(acc - avg_conf) * mask.mean()
    return ece_score

# Example usage

conf = np.array([0.9, 0.6, 0.3, 0.8])
correct = np.array([1, 0, 0, 1])
print("ECE:", ece(conf, correct))

Safety and Alignment Evaluation

Several capstone lessons add safety-focused evaluation layers critical for responsible deployment.

Refusal Evaluation Metrics

Lesson 84 in phases/19-capstone-projects/84-refusal-evaluation/docs/en.md defines under-refusal and over-refusal rates. Under-refusal occurs when unsafe prompts receive responses; over-refusal occurs when safe prompts are incorrectly rejected.

def refusal_metrics(prompts, labels, responses, threshold=0.5):
    # labels: 1 = safe, 0 = unsafe

    # responses contain "[conf=0.X]" confidence tags

    probs = np.array([float(r.split("[conf=")[1].split("]")[0]) 
                      for r in responses])
    preds = (probs >= threshold).astype(int)  # 1 = refuses

    
    under = np.mean((labels == 0) & (preds == 0))
    over = np.mean((labels == 1) & (preds == 1))
    return {"under_refusal": under, "over_refusal": over}

Dual-Use and Constitutional Evaluation

The WMDP (Weapons of Mass Destruction Proxy) benchmark in phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation/docs/en.md measures malicious-use potential across biological, cyber, and chemical domains through multiple-choice questions.

The Constitutional Rules Engine in phases/19-capstone-projects/86-constitutional-rules-engine/docs/en.md evaluates rule violations (pass/violation/not-applicable) and aggregates severity scores for policy compliance.

End-to-End RAG and Agent Metrics

When evaluating complete agent systems with retrieval, reranking, and generation components, the curriculum shifts to information-retrieval metrics. The RAG evaluation suite in phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md implements:

• Precision@k and Recall@k for retrieval quality
• MRR (Mean Reciprocal Rank) and nDCG (normalized Discounted Cumulative Gain) for ranking evaluation
• Faithfulness scores measuring answer-to-source overlap
• Answer relevance metrics for semantic alignment

Production Load Testing Tools

For scalability assessment, phases/17-infrastructure-and-production/22-load-testing-llm-apis/docs/en.md introduces streaming-aware benchmarking utilities:

LLMPerf provides token-level latency and throughput measurement under concurrent load.

guidellm generates synthetic workloads for large-scale token benchmarking, simulating real-world traffic patterns before deployment.

Summary

• Academic benchmarks (MMLU-Pro, HumanEval, MT-Bench-v2) provide standardized capability baselines for general knowledge, coding, and chat performance
• EleutherAI's lm-evaluation-harness offers a unified interface for running 200+ tasks with consistent metric aggregation
• Calibration metrics (ECE) and perplexity reveal model confidence characteristics invisible to accuracy scores alone
• Safety evaluation requires specialized frameworks measuring refusal rates, dual-use potential (WMDP), and constitutional compliance
• RAG systems demand information-retrieval metrics (Precision@k, Recall@k, MRR) alongside generative quality measures
• Production testing utilizes LLMPerf and guidellm for load testing and latency benchmarking under realistic conditions

Frequently Asked Questions

What is the lm-evaluation-harness and why is it preferred over running benchmarks manually?

EleutherAI's lm-evaluation-harness is an open-source framework that standardizes the execution of over 200 evaluation tasks through a uniform API. According to the repository's LLM engineering phase documentation at phases/11-llm-engineering/10-evaluation/docs/en.md, it eliminates metric implementation errors and ensures comparable results across different model architectures by handling tokenization, prompt formatting, and statistical aggregation automatically.

How does Expected Calibration Error (ECE) help in production LLM systems?

ECE measures the alignment between a model's predicted confidence scores and its actual accuracy. As implemented in phases/19-capstone-projects/73-perplexity-calibration/docs/en.md, this metric identifies models that are "confidently wrong"—a critical failure mode for high-stakes applications. Well-calibrated models enable better uncertainty quantification, allowing downstream systems to trigger human oversight or fallback behaviors when confidence thresholds are not met.

What metrics should I use specifically for retrieval-augmented generation (RAG) pipelines?

For RAG systems, evaluate both the retrieval and generation components separately. The curriculum in phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md recommends Precision@k and Recall@k for retrieval accuracy, nDCG for ranking quality, and faithfulness scores to verify that generated answers stem from retrieved documents rather than hallucinated content.

How do I evaluate safety and refusal behavior in aligned language models?

Safety evaluation requires measuring under-refusal rates (failing to reject harmful prompts) and over-refusal rates (incorrectly rejecting benign prompts), as detailed in phases/19-capstone-projects/84-refusal-evaluation/docs/en.md. Additionally, the WMDP benchmark in phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation/docs/en.md provides a standardized "yellow-zone" test for potentially dangerous knowledge in chemical, biological, and cyber domains.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}