# Evaluation Metrics and Benchmarking Approaches in the AI Engineering Curriculum

> Learn AI engineering evaluation metrics and benchmarking. Explore twelve metric families for reproducible AI system evaluation across NLP RAG multimodal and safety critical apps.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: tutorial
- Published: 2026-07-30

---

**The AI Engineering from Scratch curriculum teaches a systematic, first-principles approach to evaluating AI systems through twelve distinct metric families, unified by a JSONL task specification and a centralized metric dispatcher that ensures reproducible benchmarking across classical NLP, RAG, multimodal, and safety-critical applications.**

The **rohitg00/ai-engineering-from-scratch** repository implements these **evaluation metrics and benchmarking approaches** across Phase 19 capstone projects, progressing from surface-level generation metrics to deep calibration diagnostics and safety alignment checks. Each lesson provides runnable code and strict schema contracts, enabling practitioners to build production-grade evaluation pipelines rather than relying on ad-hoc scripts.

## Core Metric Families

The curriculum organizes evaluation into foundational categories spanning classical language generation, calibration diagnostics, retrieval systems, and safety alignment.

### Classical Language Generation Metrics

Lesson 71 ([`phases/19-capstone-projects/71-classical-metrics/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/71-classical-metrics/code/main.py)) implements the **unified metric dispatcher** through a `score(metric_name, prediction, targets)` function. This dispatcher routes calls to surface-overlap metrics including **BLEU-4**, **ROUGE-L**, **F1**, **Exact-Match**, and **Accuracy**. These metrics measure lexical overlap between model outputs and reference texts, serving as the baseline for machine translation, summarization, and code generation evaluation.

### Calibration and Confidence Metrics

Lesson 73 ([`phases/19-capstone-projects/73-perplexity-calibration/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/73-perplexity-calibration/docs/en.md)) introduces **Expected Calibration Error (ECE)**, **Brier score**, and **Perplexity**. These metrics evaluate how well model confidence aligns with actual correctness, critical for model selection in safety-critical deployments. The lesson implements a calibration layer that gathers `(confidence, correct)` pairs across evaluation runs to compute these diagnostics.

### RAG and Retrieval Metrics

Lesson 68 ([`phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md)) covers retrieval-focused metrics including **Precision@k**, **Recall@k**, **MRR** (Mean Reciprocal Rank), and **nDCG**. These are paired with answer-quality metrics such as **Faithfulness** and **Answer Relevance**, creating a dual-view evaluation that grades both the retrieval component and the generation quality in RAG pipelines.

### Multimodal Evaluation Metrics

Lesson 63 ([`phases/19-capstone-projects/63-multimodal-eval/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/63-multimodal-eval/docs/en.md)) extends the framework to vision-language tasks with **Retrieval@k** for image-to-caption matching, **BLEU-4** for caption generation quality, and **Exact-Match** for Visual Question Answering (VQA) systems.

## The Unified Evaluation Architecture

Beyond individual metrics, the curriculum emphasizes architectural patterns that ensure reproducibility and composability.

### Task Specification and JSONL Schema

Lesson 70 ([`phases/19-capstone-projects/70-task-spec-format/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/70-task-spec-format/docs/en.md)) establishes a strict **JSONL task specification** schema requiring fields: `task_id`, `category`, `prompt`, `targets`, `metric_name`, and `post_process`. Unknown fields trigger validation failures, ensuring every downstream evaluator operates on a stable contract. This schema enables cross-lesson compatibility and dataset versioning.

### Leaderboard Aggregation and Normalization

Lesson 74 ([`phases/19-capstone-projects/74-leaderboard-aggregation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/74-leaderboard-aggregation/docs/en.md)) implements **metric aggregation** that normalizes heterogeneous scores to a common [0, 1] range before applying user-defined weights. This prevents high-scale metrics from dominating composite scores during model comparisons.

### End-to-End Evaluation Harness

Lesson 75 ([`phases/19-capstone-projects/75-end-to-end-eval-runner/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/75-end-to-end-eval-runner/docs/en.md)) introduces the **`EvalRun`** record pattern, which bundles raw metric values, metadata, and originating task specifications into JSONL output. This harness supports CI-driven model testing by providing a unified interface for recording per-task results.

## Safety and Alignment Metrics

The curriculum dedicates specific lessons to evaluating model behavior on safety-critical dimensions.

### Refusal and Helpfulness Metrics

Lesson 84 ([`phases/19-capstone-projects/84-refusal-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/84-refusal-evaluation/docs/en.md)) treats the model as a binary classifier to measure **over-refusal** and **under-refusal** rates. This dual-metric approach evaluates alignment with safety policies while preserving helpfulness on benign inputs.

### Prompt Injection Detection

Lesson 83 ([`phases/19-capstone-projects/83-prompt-injection-detector/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/83-prompt-injection-detector/docs/en.md)) implements per-category **Precision**, **Recall**, and **F1** scores for detecting malicious prompt patterns. These metrics power guardrails for conversational agents by quantifying detection performance without degrading benign user experience.

### Constitutional Rules and Safety Gates

Lesson 86 ([`phases/19-capstone-projects/86-constitutional-rules-engine/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/86-constitutional-rules-engine/docs/en.md)) tracks per-rule violation rates against constitutional rules, enabling continuous monitoring of policy compliance.

## Operational Diagnostics

Lesson 81 ([`phases/19-capstone-projects/81-end-to-end-distributed-train/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/81-end-to-end-distributed-train/docs/en.md)) extends evaluation to training dynamics, exporting **loss**, **gradient norm**, and **step time** metrics as JSONL for post-hoc visualization. These diagnostics support scaling research and performance debugging in distributed training environments.

## Implementation Examples

The following examples demonstrate the curriculum's hands-on approach to metric computation.

### Dispatching Metrics with score()

Use the unified dispatcher to compute Exact-Match on a single-reference task:

```python
from pathlib import Path
import json

# Load a task specification (JSONL line)

task = json.loads(Path("example_task.jsonl").read_text())

pred = "The capital of France is Paris."
targets = ["Paris"]  # single-reference for exact_match

# Dispatch on metric name defined in the spec

score = score(task["metric_name"], pred, targets)
print(f"{task['metric_name']} = {score:.3f}")

# → exact_match = 1.000

```

This implementation in [`phases/19-capstone-projects/71-classical-metrics/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/71-classical-metrics/code/main.py) routes calls based on a closed metric vocabulary defined in Lesson 70.

### Aggregating Leaderboard Results

Combine heterogeneous metrics using normalized weighting:

```python
from leaderboard import aggregate  # pseudo-module from Lesson 74

# Two metric results for the same model

results = [
    {"metric_name": "bleu_4", "value": 0.42},
    {"metric_name": "accuracy", "value": 0.88},
]

# Normalize to [0,1] and compute weighted mean

overall = aggregate(results, weights={"bleu_4": 0.5, "accuracy": 0.5})
print(f"Overall score: {overall:.3f}")

# → Overall score: 0.65

```

### Computing Calibration Metrics

Analyze model confidence alignment:

```python
from calibration import compute_calibration  # from Lesson 73

# (confidence, correct) pairs from eval runs

pairs = [(0.9, True), (0.6, False), (0.3, False), (0.8, True)]

ece, brier = compute_calibration(pairs)
print(f"ECE={ece:.2%}, Brier={brier:.3f}")

# → ECE=12.50%, Brier=0.195

```

## Summary

The AI Engineering from Scratch curriculum provides a **complete metric stack** for production AI evaluation:

- **Twelve metric families** covering classical NLP, RAG, multimodal, calibration, and safety dimensions
- **Unified dispatcher** architecture via `score()` in Lesson 71 with closed vocabulary validation
- **Strict JSONL contracts** in Lesson 70 ensuring reproducible task specifications
- **Normalization and weighting** via Leaderboard Aggregation (Lesson 74) for fair model comparisons
- **Calibration diagnostics** (ECE, Brier) in Lesson 73 for confidence alignment
- **Safety-centric evaluation** including refusal metrics (Lesson 84) and prompt injection detection (Lesson 83)
- **End-to-end harness** with `EvalRun` records (Lesson 75) for CI integration

## Frequently Asked Questions

### What is the unified metric dispatcher in the AI Engineering curriculum?

The **unified metric dispatcher** is a single `score(metric_name, prediction, targets)` function implemented in [`phases/19-capstone-projects/71-classical-metrics/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/71-classical-metrics/code/main.py). It routes evaluation calls to the appropriate metric implementation based on a closed vocabulary defined in the JSONL task specification, ensuring consistent metric computation across different model types and tasks.

### How does the curriculum handle comparing models with different metric scales?

Lesson 74 ([`phases/19-capstone-projects/74-leaderboard-aggregation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/74-leaderboard-aggregation/docs/en.md)) implements **leaderboard aggregation** that normalizes all metrics to a common [0, 1] range before applying optional user-defined weights. This prevents metrics with naturally higher scales (like accuracy versus BLEU) from unfairly dominating composite scores during model comparisons.

### What safety-specific evaluation metrics are taught?

The curriculum teaches **refusal evaluation** (Lesson 84) to measure over-refusal and under-refusal rates on safety-labeled prompts, and **prompt injection detection** (Lesson 83) using per-category precision, recall, and F1 scores. Additionally, Lesson 86 introduces constitutional rules violation tracking to monitor policy compliance rates.

### Why does the curriculum emphasize JSONL task specifications?

The strict **JSONL task specification** schema (Lesson 70) guarantees that every evaluator receives identical input fields (`task_id`, `category`, `prompt`, `targets`, `metric_name`, `post_process`). This contract ensures cross-lesson compatibility, prevents silent failures from missing fields, and enables reproducible research by versioning datasets alongside their evaluation protocols.