# LLM and Agent Evaluation Frameworks and Benchmarks: A Complete Guide from ai-engineering-from-scratch

> Master LLM and agent evaluation with a complete guide covering benchmarks like MMLU-Pro, HumanEval, and safety metrics. Learn production-grade assessment strategies for robust AI systems.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: tutorial
- Published: 2026-06-23

---

**The curriculum teaches a layered evaluation strategy spanning from classic academic benchmarks like MMLU-Pro and HumanEval to production-grade harnesses, calibration metrics, and safety-specific benchmarks for assessing both base models and agent systems.**

This guide distills the complete evaluation methodology taught in the `rohitg00/ai-engineering-from-scratch` repository. Whether you are benchmarking foundation models or production RAG pipelines, these evaluation frameworks provide the quantitative rigor needed to move from prototype to deployment.

## Core Academic Benchmarks for LLM Assessment

The foundational lessons introduce canonical test collections that serve as the de-facto standard for assessing language model capabilities. These benchmarks appear in [`phases/10-llms-from-scratch/10-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/10-evaluation/docs/en.md).

**MMLU-Pro** (Massive Multitask Language Understanding) evaluates knowledge across approximately 15,000 multiple-choice questions spanning 57 subjects. This benchmark serves as the primary "knowledge" metric for general intelligence assessment.

**HumanEval** tests functional correctness through 164 Python programming problems, measuring a model's ability to generate executable, logically sound code.

**MT-Bench-v2** provides a chat-oriented evaluation framework that scores helpfulness, factuality, and safety in conversational contexts. This appears in the fine-tuning pipeline documentation at [`phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md).

Additional task-specific suites include **SWE-Bench** for software engineering, **MATH** and **GSM8K** for arithmetic reasoning, and **ARC** with **HellaSwag** for commonsense inference.

## The lm-evaluation-harness Framework

Moving beyond individual benchmarks, the curriculum teaches **EleutherAI's `lm-evaluation-harness`**, a community-maintained framework aggregating over 200 tasks. This harness is introduced in [`phases/11-llm-engineering/10-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/10-evaluation/docs/en.md).

The harness provides three critical capabilities:

- **Uniform command-line interface**: Execute evaluations via `lm_eval --model <path> --tasks <list>`
- **Automatic metric aggregation**: Computes accuracy, F1, exact-match, BLEU, and ROUGE scores consistently across tasks
- **Plugin support**: Extends to custom evaluations like RAGAS for retrieval-augmented generation

```python

# Running MMLU and HumanEval via the harness

from lm_eval import LMEval

model = "EleutherAI/gpt-neo-2.7B"
tasks = ["mmlu", "humaneval"]
results = LMEval(model=model, tasks=tasks).run()
print(results)  # {'mmlu': 68.4, 'humaneval': 71.2}

```

## Calibration and Perplexity Metrics

Raw accuracy scores prove insufficient for production deployments. The curriculum teaches **Expected Calibration Error (ECE)** and **perplexity** as complementary diagnostics in [`phases/19-capstone-projects/73-perplexity-calibration/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/73-perplexity-calibration/docs/en.md).

**ECE** quantifies how well predicted probabilities align with empirical correctness. A well-calibrated model with 0.9 confidence should be correct 90% of the time.

**Perplexity** measures model confidence on held-out text corpora, offering insight into generative capabilities distinct from downstream task performance.

```python
import numpy as np

def ece(confidences, labels, n_bins=10):
    bins = np.linspace(0, 1, n_bins + 1)
    bin_idxs = np.digitize(confidences, bins) - 1
    ece_score = 0.0
    
    for b in range(n_bins):
        mask = bin_idxs == b
        if mask.sum() == 0:
            continue
        acc = labels[mask].mean()
        avg_conf = confidences[mask].mean()
        ece_score += np.abs(acc - avg_conf) * mask.mean()
    return ece_score

# Example usage

conf = np.array([0.9, 0.6, 0.3, 0.8])
correct = np.array([1, 0, 0, 1])
print("ECE:", ece(conf, correct))

```

## Safety and Alignment Evaluation

Several capstone lessons add safety-focused evaluation layers critical for responsible deployment.

### Refusal Evaluation Metrics

Lesson 84 in [`phases/19-capstone-projects/84-refusal-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/84-refusal-evaluation/docs/en.md) defines **under-refusal** and **over-refusal** rates. Under-refusal occurs when unsafe prompts receive responses; over-refusal occurs when safe prompts are incorrectly rejected.

```python
def refusal_metrics(prompts, labels, responses, threshold=0.5):
    # labels: 1 = safe, 0 = unsafe

    # responses contain "[conf=0.X]" confidence tags

    probs = np.array([float(r.split("[conf=")[1].split("]")[0]) 
                      for r in responses])
    preds = (probs >= threshold).astype(int)  # 1 = refuses

    
    under = np.mean((labels == 0) & (preds == 0))
    over = np.mean((labels == 1) & (preds == 1))
    return {"under_refusal": under, "over_refusal": over}

```

### Dual-Use and Constitutional Evaluation

The **WMDP** (Weapons of Mass Destruction Proxy) benchmark in [`phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation/docs/en.md) measures malicious-use potential across biological, cyber, and chemical domains through multiple-choice questions.

The **Constitutional Rules Engine** in [`phases/19-capstone-projects/86-constitutional-rules-engine/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/86-constitutional-rules-engine/docs/en.md) evaluates rule violations (pass/violation/not-applicable) and aggregates severity scores for policy compliance.

## End-to-End RAG and Agent Metrics

When evaluating complete agent systems with retrieval, reranking, and generation components, the curriculum shifts to information-retrieval metrics. The RAG evaluation suite in [`phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md) implements:

- **Precision@k** and **Recall@k** for retrieval quality
- **MRR** (Mean Reciprocal Rank) and **nDCG** (normalized Discounted Cumulative Gain) for ranking evaluation
- **Faithfulness** scores measuring answer-to-source overlap
- **Answer relevance** metrics for semantic alignment

## Production Load Testing Tools

For scalability assessment, [`phases/17-infrastructure-and-production/22-load-testing-llm-apis/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/17-infrastructure-and-production/22-load-testing-llm-apis/docs/en.md) introduces streaming-aware benchmarking utilities:

**LLMPerf** provides token-level latency and throughput measurement under concurrent load.

**guidellm** generates synthetic workloads for large-scale token benchmarking, simulating real-world traffic patterns before deployment.

## Summary

- **Academic benchmarks** (MMLU-Pro, HumanEval, MT-Bench-v2) provide standardized capability baselines for general knowledge, coding, and chat performance
- **EleutherAI's lm-evaluation-harness** offers a unified interface for running 200+ tasks with consistent metric aggregation
- **Calibration metrics** (ECE) and perplexity reveal model confidence characteristics invisible to accuracy scores alone
- **Safety evaluation** requires specialized frameworks measuring refusal rates, dual-use potential (WMDP), and constitutional compliance
- **RAG systems** demand information-retrieval metrics (Precision@k, Recall@k, MRR) alongside generative quality measures
- **Production testing** utilizes LLMPerf and guidellm for load testing and latency benchmarking under realistic conditions

## Frequently Asked Questions

### What is the lm-evaluation-harness and why is it preferred over running benchmarks manually?

**EleutherAI's `lm-evaluation-harness`** is an open-source framework that standardizes the execution of over 200 evaluation tasks through a uniform API. According to the repository's LLM engineering phase documentation at [`phases/11-llm-engineering/10-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/10-evaluation/docs/en.md), it eliminates metric implementation errors and ensures comparable results across different model architectures by handling tokenization, prompt formatting, and statistical aggregation automatically.

### How does Expected Calibration Error (ECE) help in production LLM systems?

**ECE** measures the alignment between a model's predicted confidence scores and its actual accuracy. As implemented in [`phases/19-capstone-projects/73-perplexity-calibration/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/73-perplexity-calibration/docs/en.md), this metric identifies models that are "confidently wrong"—a critical failure mode for high-stakes applications. Well-calibrated models enable better uncertainty quantification, allowing downstream systems to trigger human oversight or fallback behaviors when confidence thresholds are not met.

### What metrics should I use specifically for retrieval-augmented generation (RAG) pipelines?

For RAG systems, evaluate both the **retrieval** and **generation** components separately. The curriculum in [`phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md) recommends **Precision@k** and **Recall@k** for retrieval accuracy, **nDCG** for ranking quality, and **faithfulness** scores to verify that generated answers stem from retrieved documents rather than hallucinated content.

### How do I evaluate safety and refusal behavior in aligned language models?

Safety evaluation requires measuring **under-refusal rates** (failing to reject harmful prompts) and **over-refusal rates** (incorrectly rejecting benign prompts), as detailed in [`phases/19-capstone-projects/84-refusal-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/84-refusal-evaluation/docs/en.md). Additionally, the **WMDP** benchmark in [`phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation/docs/en.md) provides a standardized "yellow-zone" test for potentially dangerous knowledge in chemical, biological, and cyber domains.