LLM Evaluation Frameworks in Phase 5: RAGAS, DeepEval, and G-Eval Explained

RAGAS provides reference-free metrics for live RAG monitoring, DeepEval offers pytest-style regression testing for CI/CD pipelines, and G-Eval enables custom chain-of-thought judging for domain-specific criteria, all requiring calibration against hand-labeled data (≥50 examples, Spearman ρ > 0.7) before deployment.

Phase 5 of the rohitg00/ai-engineering-from-scratch repository covers three production-grade LLM evaluation frameworks in the lesson titled "LLM Evaluation — RAGAS, DeepEval, G-Eval." According to the source documentation in phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md, these frameworks address different stages of the RAG lifecycle: continuous monitoring, automated regression gates, and bespoke quality assessment.

RAGAS: Reference-Free RAG Metrics

RAGAS (Retrieval-Augmented Generation Assessment) is a reference-free metric library designed for evaluating RAG pipelines without requiring ground-truth answers. The framework defines four core metrics—faithfulness, answer-relevance, context-precision, and context-recall—by combining natural-language-inference (NLI) checks with LLM-as-judge patterns.

How RAGAS Works

As implemented in the lesson documentation, RAGAS extracts atomic claims from generated answers and verifies each against retrieved contexts using an NLI model. The process works as follows:

1. Claim Extraction: An LLM breaks the answer into verifiable statements.
2. NLI Verification: Each claim is checked against the context for entailment, contradiction, or neutrality.
3. Score Aggregation: The fraction of supported claims produces the faithfulness score, while context metrics measure precision and recall of retrieved chunks.

RAGAS Implementation Example

The toy implementation in phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/code/main.py demonstrates the core NLI-based faithfulness calculation:

# Example: RAGAS-style faithfulness using an NLI model

from transformers import pipeline

nli = pipeline(
    "text-classification",
    model="MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
    top_k=None,
)

def faithfulness(answer: str, context: str) -> float:
    # Break answer into atomic claims (LLM-generated)

    claims = answer.splitlines()
    if not claims:
        return 0.0
    supported = 0
    for claim in claims:
        result = nli({"text": context, "text_pair": claim})[0]
        entail = next((s for s in result if s["label"] == "entailment"), None)
        if entail and entail["score"] > 0.5:
            supported += 1
    return supported / len(claims)

DeepEval: Pytest-Style Testing Harness

DeepEval provides a testing framework that integrates directly into CI/CD pipelines. Unlike RAGAS, which targets production monitoring, DeepEval focuses on regression testing through familiar pytest-style assertions.

CI/CD Integration Pattern

The framework supplies ready-made metrics such as FaithfulnessMetric and ContextualRelevancyMetric that can be imported into unit tests. A test suite loads a golden dataset, measures each metric against defined thresholds, and blocks merges when scores degrade. According to the lesson documentation, this pattern catches regressions before deployment by asserting thresholds like faith.score >= 0.85.

DeepEval Code Example

The following pattern from the repository demonstrates a complete CI gate implementation:

# Example: DeepEval CI gate

import deepeval
from deepeval.metrics import FaithfulnessMetric, ContextualRelevancyMetric

faith = FaithfulnessMetric(threshold=0.85)
relev = ContextualRelevancyMetric(threshold=0.7)

def test_rag_regression():
    cases = load_gold_cases()          # hand-labeled QA set

    for case in cases:
        faith.measure(case)
        assert faith.score >= 0.85, f"Faithfulness regression in {case.id}"
        relev.measure(case)
        assert relev.score >= 0.7, f"Relevancy regression in {case.id}"

G-Eval: Custom LLM-as-Judge with Chain-of-Thought

G-Eval addresses domain-specific evaluation criteria that standard metrics cannot capture. The framework wraps a custom rubric—combining explicit criteria and evaluation steps—into a single metric that leverages chain-of-thought reasoning.

Chain-of-Thought Reasoning Architecture

G-Eval prompts a judge LLM to follow explicit evaluation steps, aggregate per-claim scores, and return a normalized 0-1 value. As noted in docs/en.md, the metric is implemented within DeepEval as the GEval class and can utilize any LLM as the judge model, making it suitable for specialized dimensions like citation correctness or technical accuracy.

G-Eval Implementation

The following example from the source code demonstrates defining a custom correctness metric:

# Example: G-Eval custom metric (correctness)

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

g_metric = GEval(
    name="Correctness",
    criteria="The answer should be factually accurate and match the expected output.",
    evaluation_steps=[
        "Read the expected output.",
        "Read the actual output.",
        "List factual claims in the actual output.",
        "For each claim, mark supported or unsupported by the expected output.",
        "Return score = fraction supported.",
    ],
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
)

test = LLMTestCase(
    input="When was the first iPhone released?",
    actual_output="June 29th, 2007.",
    expected_output="June 29, 2007."
)
g_metric.measure(test)
print(g_metric.score, g_metric.reason)

The Trust Layer: Calibration Requirements

All three frameworks share a calibration requirement that mitigates judge bias and model drift. Before deploying any LLM-as-judge metric, you must calibrate the judge against a hand-labeled dataset containing at least 50 examples, achieving a Spearman correlation (ρ) greater than 0.7 with human judgments.

This calibration step, emphasized in the lesson's pitfalls section, prevents JSON-parsing failures and ensures that automated scores reflect actual human quality assessments.

Production Integration Patterns

The typical production stack combines all three frameworks at different layers:

• RAGAS runs against live traffic for continuous monitoring and drift detection.
• DeepEval executes in CI/CD pipelines to block regressions before merge.
• G-Eval handles bespoke quality dimensions not covered by standard RAGAS metrics.

This architecture is documented in the sample artifact phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/outputs/skill-eval-architect.md, which outlines the complete evaluation design pattern.

Summary

• RAGAS provides four reference-free metrics (faithfulness, answer-relevance, context-precision, context-recall) using NLI verification for production RAG monitoring.
• DeepEval offers pytest-style unit testing with built-in metrics like FaithfulnessMetric for CI/CD regression gates.
• G-Eval enables custom rubrics with chain-of-thought reasoning through the GEval class for domain-specific criteria.
• All frameworks require pre-deployment calibration on ≥50 hand-labeled examples with Spearman ρ > 0.7 to ensure judge reliability.
• Source implementations reside in phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/code/main.py, with documentation in docs/en.md.

Frequently Asked Questions

What is the primary difference between RAGAS and DeepEval?

RAGAS focuses on reference-free evaluation of production RAG pipelines using NLI-based metrics without ground-truth answers, while DeepEval provides a pytest-style testing harness designed for CI/CD integration with explicit threshold assertions. RAGAS suits live monitoring, whereas DeepEval targets regression testing during development.

How does G-Eval differ from standard RAGAS metrics?

G-Eval implements custom LLM-as-judge logic with explicit chain-of-thought reasoning steps defined in a rubric, allowing evaluation of domain-specific criteria like citation accuracy. Unlike RAGAS's fixed four-metric approach, G-Eval (implemented as the GEval class in DeepEval) adapts to bespoke quality dimensions through customizable evaluation steps and criteria.

Why is calibration mandatory before deploying these evaluation frameworks?

Calibration ensures the LLM judge's scores correlate with human judgment (Spearman ρ > 0.7) on at least 50 hand-labeled examples. This step mitigates judge bias, prevents JSON-parsing failures in automated pipelines, and accounts for model drift that could invalidate metric reliability in production environments.

Which framework should I use for monitoring versus testing?

Use RAGAS for continuous monitoring of live RAG traffic to detect answer-quality degradation over time. Use DeepEval for pre-deployment regression testing in CI/CD pipelines to block code merges that reduce faithfulness or relevancy scores. Reserve G-Eval for evaluating specialized quality dimensions that RAGAS does not natively cover, such as technical correctness or stylistic compliance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}