# How the AI Engineering From Scratch Curriculum Teaches Evaluation Metrics for Machine Learning Models: A Progressive Framework

> Learn how the AI Engineering From Scratch curriculum teaches ML evaluation metrics. Progress from basic accuracy to LLM judge frameworks and bias detection with this progressive approach.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-07-31

---

**The curriculum adopts a layered, domain-specific approach to teaching evaluation metrics for machine learning models, progressing from foundational accuracy pitfalls in classic ML to advanced LLM-as-judge frameworks and ethical bias detection.**

The rohitg00/ai-engineering-from-scratch repository provides a comprehensive roadmap for mastering AI engineering, with evaluation metrics for machine learning models serving as a core pedagogical thread throughout its eighteen phases. Rather than treating model evaluation as an afterthought, the curriculum embeds metric selection deeply into each domain, teaching learners not just how to calculate scores, but why specific metrics matter for business outcomes and production reliability.

## Foundation Phase: Correcting the Accuracy Trap in Classic Machine Learning

### Handling Imbalanced Data with Precision and Recall

In [`phases/02-ml-fundamentals/17-imbalanced-data/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/02-ml-fundamentals/17-imbalanced-data/docs/en.md), the curriculum establishes the **metric-first mindset** by demonstrating why accuracy fails on skewed datasets. Learners work through hands-on exercises using the `skill-imbalanced-data` output to compare accuracy against balanced alternatives including **precision**, **recall**, and **F1 score**. The lesson emphasizes that metric selection must precede model architecture decisions when classes are unevenly distributed.

## Domain-Specific Metric Frameworks

### NLP Evaluation: From Sentiment to Coreference

The Natural Language Processing track in Phase 05 introduces specialized metrics tailored to linguistic tasks. In the sentiment analysis lesson, learners design **custom evaluation metrics that matter** for business goals, moving beyond generic accuracy to polarity-specific measures and confusion matrix analysis.

For coreference resolution in lesson 24, the curriculum dives deep into the [`phases/05-nlp-foundations-to-advanced/24-coreference-resolution/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/05-nlp-foundations-to-advanced/24-coreference-resolution/docs/en.md) file, teaching multiple complementary metrics: **MUC**, **B³**, **CEAF**, **BLANC**, and **LEA**. Learners calculate the **CoNLL F1** aggregate score to report holistic model performance, understanding why no single metric captures all aspects of entity resolution.

### Computer Vision: Quantifying Generative Quality

Computer vision evaluation in Phase 04 addresses the unique challenges of generative models. In [`phases/04-computer-vision/09-image-generation-gans/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/04-computer-vision/09-image-generation-gans/docs/en.md), the curriculum teaches **Fréchet Inception Distance (FID)** as the de-facto benchmark for GAN evaluation. The lesson covers the conceptual foundation of distribution distance metrics and provides practical calculation workflows for assessing synthetic image quality against real data distributions.

### Speech and Audio: Signal-Based Assessment

Phase 06 extends evaluation to waveform data in [`phases/06-speech-and-audio/17-audio-evaluation-metrics/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/06-speech-and-audio/17-audio-evaluation-metrics/docs/en.md). The curriculum introduces domain-specific measures including **signal-to-noise ratio**, **word-error-rate**, and **anti-spoofing scores** through the `skill-audio-evaluator` artifact. This hands-on tool demonstrates metric computation on sample audio files, bridging the gap between theoretical signal processing and production-ready quality control.

## LLM Engineering: Modern Evaluation Architecture

### The Six-Step Evaluation Pipeline

The most comprehensive treatment of evaluation metrics for machine learning models appears in Phase 11's LLM engineering lessons. According to [`phases/11-llm-engineering/10-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/10-evaluation/docs/en.md), learners implement a rigorous **six-step pipeline**: dataset creation → rubric design → LLM-as-judge scorer implementation → automated metric integration → CI/CD deployment → statistical reporting.

This framework introduces **LLM-as-Judge** methodologies where strong language models score outputs against anchored rubrics, alongside traditional automated metrics like **ROUGE-L** and semantic similarity scores. The curriculum emphasizes statistical rigor through confidence intervals and sample-size calculations, ensuring learners understand the reliability of their evaluations.

### Production Tool Integration

The curriculum provides runnable implementations using industry-standard tools. The `promptfoo` configuration in the evaluation lesson demonstrates YAML-based test suites:

```yaml

# File: promptfoo.yaml

providers:
  - openai:gpt-4o
tests:
  - vars:
      question: "What is the capital of France?"
    assert:
      - type: contains
        value: "Paris"
      - type: llm-rubric
        value: "The answer must be factually correct, concise, and mention the city name."

```

Running `promptfoo eval` generates a JSON report with pass/fail status and rubric scores. For Python-based verification, the curriculum implements **DeepEval** metrics:

```python
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="Paris is the capital of France.",
    expected_output="Paris",
    retrieval_context=["France is a European country."]
)

relevancy = AnswerRelevancyMetric(threshold=0.7)
faithfulness = FaithfulnessMetric(threshold=0.7)

print("Relevancy:", relevancy.verify(test_case))
print("Faithfulness:", faithfulness.verify(test_case))

```

### Reusable Evaluation Patterns

Beyond implementation, [`phases/11-llm-engineering/10-evaluation/outputs/skill-eval-patterns.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/10-evaluation/outputs/skill-eval-patterns.md) provides a decision framework for metric selection. The **skill-eval-patterns** artifact guides learners through **tiered evaluation strategies**: cheap first-pass automated checks, LLM-as-judge validation, and human-in-the-loop verification for cost-effective quality assurance.

The [`phases/11-llm-engineering/10-evaluation/outputs/prompt-eval-designer.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/10-evaluation/outputs/prompt-eval-designer.md) template further accelerates development by converting natural-language descriptions into concrete evaluation plans complete with criteria, rubrics, and test suites.

## Ethical Evaluation and Bias Detection

Phase 18 addresses fairness in [`phases/18-ethics-safety-alignment/20-bias-representational-harm/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/18-ethics-safety-alignment/20-bias-representational-harm/docs/en.md), categorizing metrics as **embedding-based**, **probability-based**, or **generated-text-based**. The curriculum connects technical metric selection to ethical responsibilities, teaching learners to detect representational harm through specialized fairness metrics alongside traditional performance measures.

## Summary

- The curriculum teaches **evaluation metrics for machine learning models** through progressive complexity, starting with classic supervised learning measures in Phase 02 and advancing to LLM-specific rubrics in Phase 11.
- Each domain (NLP, vision, audio, LLMs) receives specialized metric treatment rather than one-size-fits-all accuracy scores, including **CoNLL F1**, **FID**, and **word-error-rate**.
- Hands-on artifacts including [`skill-eval-patterns.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/skill-eval-patterns.md), [`prompt-eval-designer.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/prompt-eval-designer.md), and `skill-audio-evaluator` provide reusable templates for production evaluation pipelines.
- The Phase 11 LLM lessons establish a complete six-step evaluation framework integrating statistical rigor, automated metrics like **ROUGE-L**, and **LLM-as-judge** methodologies.
- Tool-specific implementations for **promptfoo** and **DeepEval** demonstrate practical CI/CD integration for continuous model evaluation.

## Frequently Asked Questions

### What makes the AI Engineering From Scratch approach to evaluation metrics different from standard ML courses?

Unlike courses that treat evaluation as a final step, this curriculum embeds **metric-first thinking** into every phase. Starting with [`phases/02-ml-fundamentals/17-imbalanced-data/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/02-ml-fundamentals/17-imbalanced-data/docs/en.md), learners discover why accuracy fails before building models, while Phase 11's six-step pipeline treats evaluation design as a prerequisite for LLM deployment rather than an afterthought.

### Which evaluation metrics does the curriculum recommend for large language models?

According to [`phases/11-llm-engineering/10-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/10-evaluation/docs/en.md), the curriculum advocates for a layered approach: **ROUGE-L** and similarity scores for automated filtering, **LLM-as-judge** for nuanced quality assessment against rubrics, and human validation for critical decisions. The [`skill-eval-patterns.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/skill-eval-patterns.md) artifact provides a decision tree for matching use cases to these metric families.

### How does the curriculum teach the calculation of domain-specific metrics like FID or CoNLL F1?

For **Fréchet Inception Distance (FID)** in computer vision, the GAN lesson in Phase 04 provides conceptual explanations of distribution distance alongside practical calculation tips. For **CoNLL F1** in coreference resolution, Phase 05 breaks down the MUC, B³, CEAF, BLANC, and LEA components before demonstrating the averaging formula, ensuring learners understand both individual metric semantics and aggregate reporting.

### Does the curriculum cover statistical significance in model evaluation?

Yes, the LLM evaluation lesson in [`phases/11-llm-engineering/10-evaluation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/10-evaluation/docs/en.md) explicitly teaches statistical rigor including confidence interval calculation and sample-size formulas. This foundation ensures learners can distinguish between meaningful performance improvements and random variation when comparing evaluation metrics across model iterations.