# LLM Evaluation Frameworks in Phase 5: RAGAS, DeepEval, and G-Eval Explained

> Explore LLM evaluation frameworks like RAGAS, DeepEval, and G-Eval. Learn how to calibrate RAGAS for live monitoring, DeepEval for CI/CD testing, and G-Eval for custom judging to ensure robust AI deployment.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-08-31

---

**RAGAS provides reference-free metrics for live RAG monitoring, DeepEval offers pytest-style regression testing for CI/CD pipelines, and G-Eval enables custom chain-of-thought judging for domain-specific criteria, all requiring calibration against hand-labeled data (≥50 examples, Spearman ρ > 0.7) before deployment.**

Phase 5 of the `rohitg00/ai-engineering-from-scratch` repository covers three production-grade LLM evaluation frameworks in the lesson titled "LLM Evaluation — RAGAS, DeepEval, G-Eval." According to the source documentation in [`phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md), these frameworks address different stages of the RAG lifecycle: continuous monitoring, automated regression gates, and bespoke quality assessment.

## RAGAS: Reference-Free RAG Metrics

**RAGAS** (Retrieval-Augmented Generation Assessment) is a reference-free metric library designed for evaluating RAG pipelines without requiring ground-truth answers. The framework defines four core metrics—**faithfulness**, **answer-relevance**, **context-precision**, and **context-recall**—by combining natural-language-inference (NLI) checks with LLM-as-judge patterns.

### How RAGAS Works

As implemented in the lesson documentation, RAGAS extracts atomic claims from generated answers and verifies each against retrieved contexts using an NLI model. The process works as follows:

1. **Claim Extraction**: An LLM breaks the answer into verifiable statements.
2. **NLI Verification**: Each claim is checked against the context for entailment, contradiction, or neutrality.
3. **Score Aggregation**: The fraction of supported claims produces the faithfulness score, while context metrics measure precision and recall of retrieved chunks.

### RAGAS Implementation Example

The toy implementation in [`phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/code/main.py) demonstrates the core NLI-based faithfulness calculation:

```python

# Example: RAGAS-style faithfulness using an NLI model

from transformers import pipeline

nli = pipeline(
    "text-classification",
    model="MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli",
    top_k=None,
)

def faithfulness(answer: str, context: str) -> float:
    # Break answer into atomic claims (LLM-generated)

    claims = answer.splitlines()
    if not claims:
        return 0.0
    supported = 0
    for claim in claims:
        result = nli({"text": context, "text_pair": claim})[0]
        entail = next((s for s in result if s["label"] == "entailment"), None)
        if entail and entail["score"] > 0.5:
            supported += 1
    return supported / len(claims)

```

## DeepEval: Pytest-Style Testing Harness

**DeepEval** provides a testing framework that integrates directly into CI/CD pipelines. Unlike RAGAS, which targets production monitoring, DeepEval focuses on **regression testing** through familiar pytest-style assertions.

### CI/CD Integration Pattern

The framework supplies ready-made metrics such as `FaithfulnessMetric` and `ContextualRelevancyMetric` that can be imported into unit tests. A test suite loads a golden dataset, measures each metric against defined thresholds, and blocks merges when scores degrade. According to the lesson documentation, this pattern catches regressions before deployment by asserting thresholds like `faith.score >= 0.85`.

### DeepEval Code Example

The following pattern from the repository demonstrates a complete CI gate implementation:

```python

# Example: DeepEval CI gate

import deepeval
from deepeval.metrics import FaithfulnessMetric, ContextualRelevancyMetric

faith = FaithfulnessMetric(threshold=0.85)
relev = ContextualRelevancyMetric(threshold=0.7)

def test_rag_regression():
    cases = load_gold_cases()          # hand-labeled QA set

    for case in cases:
        faith.measure(case)
        assert faith.score >= 0.85, f"Faithfulness regression in {case.id}"
        relev.measure(case)
        assert relev.score >= 0.7, f"Relevancy regression in {case.id}"

```

## G-Eval: Custom LLM-as-Judge with Chain-of-Thought

**G-Eval** addresses domain-specific evaluation criteria that standard metrics cannot capture. The framework wraps a custom rubric—combining explicit criteria and evaluation steps—into a single metric that leverages chain-of-thought reasoning.

### Chain-of-Thought Reasoning Architecture

G-Eval prompts a judge LLM to follow explicit evaluation steps, aggregate per-claim scores, and return a normalized 0-1 value. As noted in [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md), the metric is implemented within DeepEval as the `GEval` class and can utilize any LLM as the judge model, making it suitable for specialized dimensions like citation correctness or technical accuracy.

### G-Eval Implementation

The following example from the source code demonstrates defining a custom correctness metric:

```python

# Example: G-Eval custom metric (correctness)

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

g_metric = GEval(
    name="Correctness",
    criteria="The answer should be factually accurate and match the expected output.",
    evaluation_steps=[
        "Read the expected output.",
        "Read the actual output.",
        "List factual claims in the actual output.",
        "For each claim, mark supported or unsupported by the expected output.",
        "Return score = fraction supported.",
    ],
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
)

test = LLMTestCase(
    input="When was the first iPhone released?",
    actual_output="June 29th, 2007.",
    expected_output="June 29, 2007."
)
g_metric.measure(test)
print(g_metric.score, g_metric.reason)

```

## The Trust Layer: Calibration Requirements

All three frameworks share a **calibration requirement** that mitigates judge bias and model drift. Before deploying any LLM-as-judge metric, you must calibrate the judge against a hand-labeled dataset containing at least 50 examples, achieving a Spearman correlation (ρ) greater than 0.7 with human judgments.

This calibration step, emphasized in the lesson's pitfalls section, prevents JSON-parsing failures and ensures that automated scores reflect actual human quality assessments.

## Production Integration Patterns

The typical production stack combines all three frameworks at different layers:

- **RAGAS** runs against live traffic for continuous monitoring and drift detection.
- **DeepEval** executes in CI/CD pipelines to block regressions before merge.
- **G-Eval** handles bespoke quality dimensions not covered by standard RAGAS metrics.

This architecture is documented in the sample artifact [`phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/outputs/skill-eval-architect.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/outputs/skill-eval-architect.md), which outlines the complete evaluation design pattern.

## Summary

- **RAGAS** provides four reference-free metrics (faithfulness, answer-relevance, context-precision, context-recall) using NLI verification for production RAG monitoring.
- **DeepEval** offers pytest-style unit testing with built-in metrics like `FaithfulnessMetric` for CI/CD regression gates.
- **G-Eval** enables custom rubrics with chain-of-thought reasoning through the `GEval` class for domain-specific criteria.
- All frameworks require pre-deployment calibration on ≥50 hand-labeled examples with Spearman ρ > 0.7 to ensure judge reliability.
- Source implementations reside in [`phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/code/main.py), with documentation in [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md).

## Frequently Asked Questions

### What is the primary difference between RAGAS and DeepEval?

RAGAS focuses on reference-free evaluation of production RAG pipelines using NLI-based metrics without ground-truth answers, while DeepEval provides a pytest-style testing harness designed for CI/CD integration with explicit threshold assertions. RAGAS suits live monitoring, whereas DeepEval targets regression testing during development.

### How does G-Eval differ from standard RAGAS metrics?

G-Eval implements custom **LLM-as-judge** logic with explicit chain-of-thought reasoning steps defined in a rubric, allowing evaluation of domain-specific criteria like citation accuracy. Unlike RAGAS's fixed four-metric approach, G-Eval (implemented as the `GEval` class in DeepEval) adapts to bespoke quality dimensions through customizable evaluation steps and criteria.

### Why is calibration mandatory before deploying these evaluation frameworks?

Calibration ensures the LLM judge's scores correlate with human judgment (Spearman ρ > 0.7) on at least 50 hand-labeled examples. This step mitigates **judge bias**, prevents JSON-parsing failures in automated pipelines, and accounts for model drift that could invalidate metric reliability in production environments.

### Which framework should I use for monitoring versus testing?

Use **RAGAS** for continuous monitoring of live RAG traffic to detect answer-quality degradation over time. Use **DeepEval** for pre-deployment regression testing in CI/CD pipelines to block code merges that reduce faithfulness or relevancy scores. Reserve **G-Eval** for evaluating specialized quality dimensions that RAGAS does not natively cover, such as technical correctness or stylistic compliance.