LLM Evaluation Frameworks in AI Engineering from Scratch: RAGAS, DeepEval, and G-Eval Explained

The AI Engineering from Scratch curriculum teaches three core LLM evaluation frameworks—RAGAS for production monitoring, DeepEval for CI/CD regression testing, and G-Eval for custom domain-specific scoring—to build robust Retrieval-Augmented Generation pipelines.

The rohitg00/ai-engineering-from-scratch repository provides a comprehensive guide to evaluating large language model systems through specialized frameworks. Understanding these LLM evaluation frameworks is essential for shipping production-grade RAG pipelines that resist hallucinations and maintain quality over time. The course materials are located in phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md.

RAGAS: Reference-Free Monitoring for RAG Pipelines

RAGAS (Retrieval-Augmented Generation Assessment) serves as the primary framework for continuous production monitoring without requiring gold-standard reference answers. According to the source code analysis, this framework evaluates retrieval quality using four core metrics built on Natural Language Inference (NLI) and LLM-judge backends: faithfulness, answer-relevance, context-precision, and context-recall.

The framework specifically targets hallucination detection through its faithfulness metric, which verifies whether generated answers are grounded in retrieved context.

Implementing the Faithfulness Metric

The lesson provides a Python implementation that uses an NLI pipeline to verify atomic claims against context:

from typing import Callable
from transformers import pipeline

# NLI pipeline (any MNLI-compatible model works)

nli = pipeline(
    "text-classification",
    model="MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
    top_k=None,
)

LLM = Callable[[str], str]   # a callable that returns generated text

def atomic_claims(answer: str, llm: LLM) -> list[str]:
    prompt = f"Break this answer into simple factual claims (one per line):\n{answer}"
    return llm(prompt).splitlines()

def ragas_faithfulness(answer: str, context: str, llm: LLM) -> float:
    claims = atomic_claims(answer, llm)
    if not claims:
        return 0.0
    supported = sum(
        1
        for claim in claims
        if (result := nli({"text": context, "text_pair": claim})[0])
        and (entail := next((s for s in result if s["label"] == "entailment"), None))
        and entail["score"] > 0.5
    )
    return supported / len(claims)

Source: phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md

DeepEval: CI/CD-Native Regression Testing

DeepEval functions as "pytest for LLMs," providing 14+ built-in metrics including hallucination detection, bias checking, and toxicity analysis. As implemented in the course materials, this framework integrates directly into continuous integration pipelines to catch performance regressions before production deployment.

The framework automatically scores test suites for hallucination and jailbreak attempts, making it ideal for automated gates in software development workflows.

pytest Integration Example

The course demonstrates DeepEval integration using standard pytest patterns:

# test_evals.py

from deepeval import test_case, metrics

# Example: Hallucination metric (built-in)

test = test_case.LLMTestCase(
    input="What is the capital of France?",
    actual_output="Paris is the capital of France.",
    expected_output="Paris"
)

# Run all metrics defined in DeepEval's default suite

result = metrics.run_all(test)

assert result.overall_score > 0.9   # CI gate: fail if regression occurs

Source: phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md

G-Eval: Custom Domain-Specific Scoring

G-Eval extends the evaluation stack with LLM-as-judge capabilities for custom criteria that standard metrics cannot capture. This framework utilizes chain-of-thought evaluation to produce normalized 0-1 scores based on user-defined rubrics, enabling domain-specific checks such as citation accuracy or compliance verification.

The course positions G-Eval as a complementary tool to DeepEval, filling gaps where pre-built metrics fall short.

Defining Custom Evaluation Criteria

The lesson includes a practical example for verifying citation accuracy:

from deepeval.metrics import GEval

# Define a custom rubric

criterion = GEval(
    name="Citation Accuracy",
    criteria="The answer must cite the correct source document.",
    evaluation_steps=[
        "Read the expected source reference.",
        "Read the actual answer.",
        "Check whether the cited source matches.",
    ],
    evaluation_params=[
        "INPUT", "ACTUAL_OUTPUT", "EXPECTED_OUTPUT"
    ],
)

test = test_case.LLMTestCase(
    input="Explain the theory of relativity.",
    actual_output="Einstein's theory, see *Relativity.pdf*.",
    expected_output="Einstein's theory, see *Relativity.pdf*."
)

criterion.measure(test)
print("Score:", criterion.score)   # 0-1 score

Source: phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md

The Production Evaluation Stack

The curriculum presents a layered architecture that combines all three frameworks for comprehensive coverage:

1. RAGAS operates continuously in production environments to monitor retrieval quality and flag hallucinations via the faithfulness metric without requiring reference answers.

2. DeepEval executes within CI/CD pipelines (e.g., pytest gates) to prevent regressions from reaching production.

3. G-Eval extends DeepEval with custom rubrics, enabling specialized checks like "Did the answer cite the correct source?" or industry-specific compliance validation.

This three-tier approach ensures that monitoring, regression testing, and custom quality checks are all addressed within the evaluation pipeline.

Summary

• RAGAS provides reference-free monitoring of RAG pipelines using NLI-based metrics for faithfulness, answer-relevance, context-precision, and context-recall.
• DeepEval offers CI/CD-native regression testing with 14+ built-in metrics and pytest integration for automated quality gates.
• G-Eval enables custom domain-specific scoring through LLM-as-judge evaluation with chain-of-thought reasoning and 0-1 score outputs.
• All three frameworks are documented in phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md with accompanying quiz questions in quiz.json.
• The course teaches a layered production stack combining continuous monitoring, regression prevention, and custom criteria validation.

Frequently Asked Questions

What is the difference between RAGAS and DeepEval?

RAGAS is designed for production monitoring without reference answers, using NLI-based metrics to detect hallucinations in real-time. DeepEval is built for CI/CD integration, functioning like pytest for LLMs to catch regressions before deployment through automated testing suites.

Can G-Eval be used independently of DeepEval?

While G-Eval can operate as a standalone evaluation method, the course teaches it as a complementary extension to DeepEval. In practice, G-Eval custom criteria fill gaps where DeepEval's 14+ built-in metrics do not cover domain-specific requirements, creating a comprehensive evaluation pipeline.

How does the faithfulness metric in RAGAS detect hallucinations?

The faithfulness metric breaks generated answers into atomic claims using an LLM, then verifies each claim against retrieved context using an NLI (Natural Language Inference) model. Claims that do not entail from the context are flagged as hallucinations, producing a normalized faithfulness score between 0 and 1.

Where are these frameworks documented in the course repository?

All three LLM evaluation frameworks are covered in phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md, with knowledge verification available through phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/quiz.json. The lesson also includes visual summaries in the assets directory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}