How Document Grading Evaluates Semantic Relevance in Agentic RAG

Document grading in agentic RAG uses a structured LLM prompt to return a binary yes/no relevance score, routing the workflow to generate an answer when retrieved documents semantically match the query or to rewrite the query when they do not.

In the jamwithai/production-agentic-rag-course repository, the document grading node serves as the semantic gatekeeper of the agentic Retrieval-Augmented Generation (RAG) pipeline. This component evaluates whether text retrieved from the vector store actually answers the user’s intent, not merely whether it shares keywords. According to the source code, the grading decision drives the next graph transition—either to generate_answer when content is relevant or to rewrite_query when the retrieval misses the mark.

The Three Pillars of Semantic Evaluation

The grading system relies on three tightly coupled components that transform lexical retrieval into semantic judgment.

Handcrafted Prompt Template for Semantic Context

The GRADE_DOCUMENTS_PROMPT in src/services/agents/prompts.py instructs the LLM to look for meaningful conceptual overlap rather than surface-level token matching. The template explicitly asks the model to assess whether the retrieved documents contain information that answers the user question, providing space for reasoning alongside a binary score.

Strict Output Validation with Pydantic

To ensure deterministic parsing, the codebase defines the GradeDocuments schema in src/services/agents/models.py. This Pydantic model enforces a JSON structure containing binary_score (yes or no) and reasoning fields. The downstream GradingResult model then stores the final verdict with strong typing, mapping the binary decision to a float score (1.0 for relevant, 0.0 for irrelevant).

The Grading Node Implementation

The ainvoke_grade_documents_step function in src/services/agents/nodes/grade_documents_node.py orchestrates the entire evaluation. This async node pulls the latest user query and concatenated document context, formats the prompt, invokes the LLM with structured output constraints, and interprets the response to emit a routing decision.

Step-by-Step Semantic Grading Workflow

The document grading node executes a precise sequence to evaluate semantic relevance:

1. Extract conversation state – The node retrieves the most recent user query and retrieved context using get_latest_query(state["messages"]) and get_latest_context(state["messages"]) from the agent’s message history.

2. Format the evaluation prompt – The system interpolates the context and question into GRADE_DOCUMENTS_PROMPT.format(context=context, question=question).

3. Invoke structured LLM – Using the runtime’s Ollama client, the code wraps the LangChain model with with_structured_output(GradeDocuments), forcing JSON conformance:

llm = runtime.context.ollama_client.get_langchain_model(
    model=runtime.context.model_name, temperature=0.0
)
structured_llm = llm.with_structured_output(GradeDocuments)
grading_response = await structured_llm.ainvoke(grading_prompt)

4. Parse binary relevance – The node converts the textual binary_score to a boolean and numeric score:

is_relevant = grading_response.binary_score == "yes"
score = 1.0 if is_relevant else 0.0
grading_result = GradingResult(
    document_id="retrieved_docs",
    is_relevant=is_relevant,
    score=score,
    reasoning=grading_response.reasoning,
)

5. Determine routing – The function returns "generate_answer" when is_relevant is True, otherwise "rewrite_query".

Error Handling and Fallback Heuristics

When the LLM call fails or returns malformed output, the grading node implements a defensive fallback. As implemented in src/services/agents/nodes/grade_documents_node.py, the code catches exceptions and falls back to a length-based heuristic:

is_relevant = len(context.strip()) > 50

This ensures the agentic graph never deadlocks, defaulting to query rewriting only when the context is genuinely empty or truncated. Additionally, when Langfuse tracing is enabled, the node records spans capturing inputs, outputs, and execution timing for observability.

Practical Implementation Example

To invoke the grading node directly for testing or debugging:

import asyncio
from src.services.agents.nodes.grade_documents_node import ainvoke_grade_documents_step
from src.services.agents.state import AgentState

# Construct a mock state with user query and retrieved context

state: AgentState = {
    "messages": [
        {"role": "user", "content": "What are transformer architectures?"},
        {"role": "assistant", "content": "Research papers discussing attention mechanisms..."}
    ]
}

# Run the grading step (runtime setup omitted for brevity)

result = asyncio.run(ainvoke_grade_documents_step(state, runtime))

print(result["routing_decision"])  # "generate_answer" or "rewrite_query"

print(result["grading_results"][0].reasoning)  # Semantic explanation

Summary

• Document grading acts as the semantic filter in agentic RAG, preventing irrelevant retrievals from polluting the generation phase.
• The system uses a binary yes/no scoring mechanism via the GradeDocuments Pydantic model to ensure deterministic LLM outputs.
• Routing logic is hard-coded in ainvoke_grade_documents_step: relevant documents trigger answer generation, while irrelevant hits trigger query rewriting.
• Fallback heuristics based on text length prevent workflow failures when the LLM is unavailable.
• All components reside in src/services/agents/nodes/grade_documents_node.py, src/services/agents/prompts.py, and src/services/agents/models.py.

Frequently Asked Questions

What is the difference between lexical matching and semantic relevance in document grading?

Lexical matching checks for shared keywords between queries and documents, while semantic relevance evaluates whether the concepts and meanings align. The GRADE_DOCUMENTS_PROMPT explicitly instructs the LLM to assess semantic meaning rather than keyword overlap, allowing it to recognize that "attention mechanisms" and "transformer architectures" are conceptually related even if they share no identical tokens.

Why does the grading system use a binary yes/no score instead of a similarity threshold?

The binary yes/no output from the GradeDocuments schema simplifies the agentic routing logic into a deterministic boolean decision. This eliminates ambiguous middle ranges where traditional vector similarity scores might fall, ensuring the graph clearly chooses between generating an answer immediately or reformulating the query for better retrieval.

How does the system handle LLM failures during document grading?

If the structured LLM call raises an exception, the ainvoke_grade_documents_step function catches the error and applies a fallback heuristic: if the concatenated context has more than 50 characters, it assumes relevance. This safety net prevents the agentic workflow from crashing while favoring answer generation over unnecessary query rewriting when context exists.

Where is the grading prompt defined and how can it be customized?

The grading prompt is defined as the GRADE_DOCUMENTS_PROMPT constant in src/services/agents/prompts.py. Developers can modify this template to adjust the LLM’s evaluation criteria—for example, by adding instructions to penalize outdated information or to require specific entity mentions—without touching the node implementation logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}