# How Document Grading Evaluates Semantic Relevance in Agentic RAG

> Discover how document grading in agentic RAG uses structured LLM prompts for binary relevance scoring, optimizing your workflow for accurate answers or query refinement.

- Repository: [jamwithai/production-agentic-rag-course](https://github.com/jamwithai/production-agentic-rag-course)
- Tags: deep-dive
- Published: 2026-03-23

---

**Document grading in agentic RAG uses a structured LLM prompt to return a binary yes/no relevance score, routing the workflow to generate an answer when retrieved documents semantically match the query or to rewrite the query when they do not.**

In the `jamwithai/production-agentic-rag-course` repository, the **document grading** node serves as the semantic gatekeeper of the agentic Retrieval-Augmented Generation (RAG) pipeline. This component evaluates whether text retrieved from the vector store actually answers the user’s intent, not merely whether it shares keywords. According to the source code, the grading decision drives the next graph transition—either to **generate_answer** when content is relevant or to **rewrite_query** when the retrieval misses the mark.

## The Three Pillars of Semantic Evaluation

The grading system relies on three tightly coupled components that transform lexical retrieval into semantic judgment.

### Handcrafted Prompt Template for Semantic Context

The [`GRADE_DOCUMENTS_PROMPT`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py#L1-L13) in [`src/services/agents/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py) instructs the LLM to look for meaningful conceptual overlap rather than surface-level token matching. The template explicitly asks the model to assess whether the retrieved documents contain information that answers the user question, providing space for reasoning alongside a binary score.

### Strict Output Validation with Pydantic

To ensure deterministic parsing, the codebase defines the [`GradeDocuments`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py#L17-L26) schema in [`src/services/agents/models.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py). This Pydantic model enforces a JSON structure containing `binary_score` (`yes` or `no`) and `reasoning` fields. The downstream [`GradingResult`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py#L83-L96) model then stores the final verdict with strong typing, mapping the binary decision to a float score (`1.0` for relevant, `0.0` for irrelevant).

### The Grading Node Implementation

The [`ainvoke_grade_documents_step`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/nodes/grade_documents_node.py#L16-L54) function in [`src/services/agents/nodes/grade_documents_node.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/nodes/grade_documents_node.py) orchestrates the entire evaluation. This async node pulls the latest user query and concatenated document context, formats the prompt, invokes the LLM with structured output constraints, and interprets the response to emit a routing decision.

## Step-by-Step Semantic Grading Workflow

The document grading node executes a precise sequence to evaluate semantic relevance:

1. **Extract conversation state** – The node retrieves the most recent user query and retrieved context using `get_latest_query(state["messages"])` and `get_latest_context(state["messages"])` from the agent’s message history.

2. **Format the evaluation prompt** – The system interpolates the context and question into `GRADE_DOCUMENTS_PROMPT.format(context=context, question=question)`.

3. **Invoke structured LLM** – Using the runtime’s Ollama client, the code wraps the LangChain model with `with_structured_output(GradeDocuments)`, forcing JSON conformance:

```python
llm = runtime.context.ollama_client.get_langchain_model(
    model=runtime.context.model_name, temperature=0.0
)
structured_llm = llm.with_structured_output(GradeDocuments)
grading_response = await structured_llm.ainvoke(grading_prompt)

```

4. **Parse binary relevance** – The node converts the textual `binary_score` to a boolean and numeric score:

```python
is_relevant = grading_response.binary_score == "yes"
score = 1.0 if is_relevant else 0.0
grading_result = GradingResult(
    document_id="retrieved_docs",
    is_relevant=is_relevant,
    score=score,
    reasoning=grading_response.reasoning,
)

```

5. **Determine routing** – The function returns `"generate_answer"` when `is_relevant` is `True`, otherwise `"rewrite_query"`.

## Error Handling and Fallback Heuristics

When the LLM call fails or returns malformed output, the grading node implements a defensive fallback. As implemented in [`src/services/agents/nodes/grade_documents_node.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/nodes/grade_documents_node.py), the code catches exceptions and falls back to a length-based heuristic:

```python
is_relevant = len(context.strip()) > 50

```

This ensures the agentic graph never deadlocks, defaulting to query rewriting only when the context is genuinely empty or truncated. Additionally, when Langfuse tracing is enabled, the node records spans capturing inputs, outputs, and execution timing for observability.

## Practical Implementation Example

To invoke the grading node directly for testing or debugging:

```python
import asyncio
from src.services.agents.nodes.grade_documents_node import ainvoke_grade_documents_step
from src.services.agents.state import AgentState

# Construct a mock state with user query and retrieved context

state: AgentState = {
    "messages": [
        {"role": "user", "content": "What are transformer architectures?"},
        {"role": "assistant", "content": "Research papers discussing attention mechanisms..."}
    ]
}

# Run the grading step (runtime setup omitted for brevity)

result = asyncio.run(ainvoke_grade_documents_step(state, runtime))

print(result["routing_decision"])  # "generate_answer" or "rewrite_query"

print(result["grading_results"][0].reasoning)  # Semantic explanation

```

## Summary

- **Document grading** acts as the semantic filter in agentic RAG, preventing irrelevant retrievals from polluting the generation phase.
- The system uses a **binary yes/no scoring mechanism** via the `GradeDocuments` Pydantic model to ensure deterministic LLM outputs.
- **Routing logic** is hard-coded in `ainvoke_grade_documents_step`: relevant documents trigger answer generation, while irrelevant hits trigger query rewriting.
- **Fallback heuristics** based on text length prevent workflow failures when the LLM is unavailable.
- All components reside in [`src/services/agents/nodes/grade_documents_node.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/nodes/grade_documents_node.py), [`src/services/agents/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py), and [`src/services/agents/models.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py).

## Frequently Asked Questions

### What is the difference between lexical matching and semantic relevance in document grading?

Lexical matching checks for shared keywords between queries and documents, while **semantic relevance** evaluates whether the concepts and meanings align. The `GRADE_DOCUMENTS_PROMPT` explicitly instructs the LLM to assess semantic meaning rather than keyword overlap, allowing it to recognize that "attention mechanisms" and "transformer architectures" are conceptually related even if they share no identical tokens.

### Why does the grading system use a binary yes/no score instead of a similarity threshold?

The binary `yes`/`no` output from the `GradeDocuments` schema simplifies the agentic routing logic into a deterministic boolean decision. This eliminates ambiguous middle ranges where traditional vector similarity scores might fall, ensuring the graph clearly chooses between generating an answer immediately or reformulating the query for better retrieval.

### How does the system handle LLM failures during document grading?

If the structured LLM call raises an exception, the `ainvoke_grade_documents_step` function catches the error and applies a fallback heuristic: if the concatenated context has more than 50 characters, it assumes relevance. This safety net prevents the agentic workflow from crashing while favoring answer generation over unnecessary query rewriting when context exists.

### Where is the grading prompt defined and how can it be customized?

The grading prompt is defined as the `GRADE_DOCUMENTS_PROMPT` constant in [`src/services/agents/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py). Developers can modify this template to adjust the LLM’s evaluation criteria—for example, by adding instructions to penalize outdated information or to require specific entity mentions—without touching the node implementation logic.