How the AI Engineering From Scratch Curriculum Teaches Evaluation Metrics for Machine Learning Models: A Progressive Framework

The curriculum adopts a layered, domain-specific approach to teaching evaluation metrics for machine learning models, progressing from foundational accuracy pitfalls in classic ML to advanced LLM-as-judge frameworks and ethical bias detection.

The rohitg00/ai-engineering-from-scratch repository provides a comprehensive roadmap for mastering AI engineering, with evaluation metrics for machine learning models serving as a core pedagogical thread throughout its eighteen phases. Rather than treating model evaluation as an afterthought, the curriculum embeds metric selection deeply into each domain, teaching learners not just how to calculate scores, but why specific metrics matter for business outcomes and production reliability.

Foundation Phase: Correcting the Accuracy Trap in Classic Machine Learning

Handling Imbalanced Data with Precision and Recall

In phases/02-ml-fundamentals/17-imbalanced-data/docs/en.md, the curriculum establishes the metric-first mindset by demonstrating why accuracy fails on skewed datasets. Learners work through hands-on exercises using the skill-imbalanced-data output to compare accuracy against balanced alternatives including precision, recall, and F1 score. The lesson emphasizes that metric selection must precede model architecture decisions when classes are unevenly distributed.

Domain-Specific Metric Frameworks

NLP Evaluation: From Sentiment to Coreference

The Natural Language Processing track in Phase 05 introduces specialized metrics tailored to linguistic tasks. In the sentiment analysis lesson, learners design custom evaluation metrics that matter for business goals, moving beyond generic accuracy to polarity-specific measures and confusion matrix analysis.

For coreference resolution in lesson 24, the curriculum dives deep into the phases/05-nlp-foundations-to-advanced/24-coreference-resolution/docs/en.md file, teaching multiple complementary metrics: MUC, B³, CEAF, BLANC, and LEA. Learners calculate the CoNLL F1 aggregate score to report holistic model performance, understanding why no single metric captures all aspects of entity resolution.

Computer Vision: Quantifying Generative Quality

Computer vision evaluation in Phase 04 addresses the unique challenges of generative models. In phases/04-computer-vision/09-image-generation-gans/docs/en.md, the curriculum teaches Fréchet Inception Distance (FID) as the de-facto benchmark for GAN evaluation. The lesson covers the conceptual foundation of distribution distance metrics and provides practical calculation workflows for assessing synthetic image quality against real data distributions.

Speech and Audio: Signal-Based Assessment

Phase 06 extends evaluation to waveform data in phases/06-speech-and-audio/17-audio-evaluation-metrics/docs/en.md. The curriculum introduces domain-specific measures including signal-to-noise ratio, word-error-rate, and anti-spoofing scores through the skill-audio-evaluator artifact. This hands-on tool demonstrates metric computation on sample audio files, bridging the gap between theoretical signal processing and production-ready quality control.

LLM Engineering: Modern Evaluation Architecture

The Six-Step Evaluation Pipeline

The most comprehensive treatment of evaluation metrics for machine learning models appears in Phase 11's LLM engineering lessons. According to phases/11-llm-engineering/10-evaluation/docs/en.md, learners implement a rigorous six-step pipeline: dataset creation → rubric design → LLM-as-judge scorer implementation → automated metric integration → CI/CD deployment → statistical reporting.

This framework introduces LLM-as-Judge methodologies where strong language models score outputs against anchored rubrics, alongside traditional automated metrics like ROUGE-L and semantic similarity scores. The curriculum emphasizes statistical rigor through confidence intervals and sample-size calculations, ensuring learners understand the reliability of their evaluations.

Production Tool Integration

The curriculum provides runnable implementations using industry-standard tools. The promptfoo configuration in the evaluation lesson demonstrates YAML-based test suites:

# File: promptfoo.yaml

providers:
  - openai:gpt-4o
tests:
  - vars:
      question: "What is the capital of France?"
    assert:
      - type: contains
        value: "Paris"
      - type: llm-rubric
        value: "The answer must be factually correct, concise, and mention the city name."

Running promptfoo eval generates a JSON report with pass/fail status and rubric scores. For Python-based verification, the curriculum implements DeepEval metrics:

from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What is the capital of France?",
    actual_output="Paris is the capital of France.",
    expected_output="Paris",
    retrieval_context=["France is a European country."]
)

relevancy = AnswerRelevancyMetric(threshold=0.7)
faithfulness = FaithfulnessMetric(threshold=0.7)

print("Relevancy:", relevancy.verify(test_case))
print("Faithfulness:", faithfulness.verify(test_case))

Reusable Evaluation Patterns

Beyond implementation, phases/11-llm-engineering/10-evaluation/outputs/skill-eval-patterns.md provides a decision framework for metric selection. The skill-eval-patterns artifact guides learners through tiered evaluation strategies: cheap first-pass automated checks, LLM-as-judge validation, and human-in-the-loop verification for cost-effective quality assurance.

The phases/11-llm-engineering/10-evaluation/outputs/prompt-eval-designer.md template further accelerates development by converting natural-language descriptions into concrete evaluation plans complete with criteria, rubrics, and test suites.

Ethical Evaluation and Bias Detection

Phase 18 addresses fairness in phases/18-ethics-safety-alignment/20-bias-representational-harm/docs/en.md, categorizing metrics as embedding-based, probability-based, or generated-text-based. The curriculum connects technical metric selection to ethical responsibilities, teaching learners to detect representational harm through specialized fairness metrics alongside traditional performance measures.

Summary

• The curriculum teaches evaluation metrics for machine learning models through progressive complexity, starting with classic supervised learning measures in Phase 02 and advancing to LLM-specific rubrics in Phase 11.
• Each domain (NLP, vision, audio, LLMs) receives specialized metric treatment rather than one-size-fits-all accuracy scores, including CoNLL F1, FID, and word-error-rate.
• Hands-on artifacts including skill-eval-patterns.md, prompt-eval-designer.md, and skill-audio-evaluator provide reusable templates for production evaluation pipelines.
• The Phase 11 LLM lessons establish a complete six-step evaluation framework integrating statistical rigor, automated metrics like ROUGE-L, and LLM-as-judge methodologies.
• Tool-specific implementations for promptfoo and DeepEval demonstrate practical CI/CD integration for continuous model evaluation.

Frequently Asked Questions

What makes the AI Engineering From Scratch approach to evaluation metrics different from standard ML courses?

Unlike courses that treat evaluation as a final step, this curriculum embeds metric-first thinking into every phase. Starting with phases/02-ml-fundamentals/17-imbalanced-data/docs/en.md, learners discover why accuracy fails before building models, while Phase 11's six-step pipeline treats evaluation design as a prerequisite for LLM deployment rather than an afterthought.

Which evaluation metrics does the curriculum recommend for large language models?

According to phases/11-llm-engineering/10-evaluation/docs/en.md, the curriculum advocates for a layered approach: ROUGE-L and similarity scores for automated filtering, LLM-as-judge for nuanced quality assessment against rubrics, and human validation for critical decisions. The skill-eval-patterns.md artifact provides a decision tree for matching use cases to these metric families.

How does the curriculum teach the calculation of domain-specific metrics like FID or CoNLL F1?

For Fréchet Inception Distance (FID) in computer vision, the GAN lesson in Phase 04 provides conceptual explanations of distribution distance alongside practical calculation tips. For CoNLL F1 in coreference resolution, Phase 05 breaks down the MUC, B³, CEAF, BLANC, and LEA components before demonstrating the averaging formula, ensuring learners understand both individual metric semantics and aggregate reporting.

Does the curriculum cover statistical significance in model evaluation?

Yes, the LLM evaluation lesson in phases/11-llm-engineering/10-evaluation/docs/en.md explicitly teaches statistical rigor including confidence interval calculation and sample-size formulas. This foundation ensures learners can distinguish between meaningful performance improvements and random variation when comparing evaluation metrics across model iterations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}