# Guardrail Node Out-of-Domain Detection Mechanism in Production RAG

> Learn how the Guardrail node detects out-of-domain queries in production RAG. It uses an LLM to score relevance against a configurable threshold for AI research domain adherence.

- Repository: [jamwithai/production-agentic-rag-course](https://github.com/jamwithai/production-agentic-rag-course)
- Tags: internals
- Published: 2026-03-23

---

**The Guardrail node detects out-of-domain queries by invoking an LLM with a structured validation prompt to generate a relevance score between 0 and 100, then compares this score against a configurable threshold to determine if the answer remains within the AI research domain or should be rejected.**

The `jamwithai/production-agentic-rag-course` repository implements a protective layer in its agentic RAG pipeline through a specialized Guardrail node. This component prevents the system from processing or returning responses that stray outside the intended domain—specifically research papers about AI—by employing an LLM-driven scoring mechanism. Understanding this out-of-domain detection mechanism is critical for tuning safety and relevance in production RAG systems.

## LLM-Driven Relevance Scoring Architecture

The detection mechanism relies on four integrated components that evaluate answer legitimacy before downstream processing.

### Prompt-Based Validation Criteria

The node loads a dedicated guardrail validation prompt from [`src/services/agents/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py) that instructs the language model to evaluate responses against two distinct criteria:

- **Relevance**: Does the content address AI research papers?
- **Safety**: Does it contain disallowed content or off-topic information?

### Structured Output Enforcement

Rather than parsing free-text responses, the system uses the `GuardrailScoring` Pydantic model defined in [`src/services/agents/models.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py). The LLM is invoked with `with_structured_output(GuardrailScoring)`, which constrains the model to return a JSON-compatible object containing:

- `score`: An integer from 0 to 100 representing domain relevance
- `reason`: A textual explanation justifying the score

### Threshold-Based Decision Logic

After obtaining the structured response, the Guardrail node (implemented in [`src/services/agents/nodes/guardrail_node.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/nodes/guardrail_node.py)) compares the `score` against `self.graph_config.guardrail_threshold`:

```python
structured_llm = llm.with_structured_output(GuardrailScoring)
response: GuardrailScoring = structured_llm.invoke(
    {"prompt": guardrail_prompt, "input_text": answer}
)

if response.score >= self.graph_config.guardrail_threshold:
    guardrail_result = GuardrailScoring(score=response.score, reason=response.reason)
    # Continue processing

else:
    # Treat as out-of-domain / unsafe

    guardrail_result = GuardrailScoring(score=response.score, reason=response.reason)
    # Trigger fallback or abort

```

### Out-of-Domain Signal Interpretation

A **low score** (typically ≤ 30) signals that the LLM judged the answer to be unrelated to the target domain or unsafe. The accompanying `reason` field provides human-readable diagnostics, such as "The response talks about cooking recipes, not AI research."

## Key Implementation Files

The out-of-domain detection mechanism spans several critical files in the repository:

- **[`src/services/agents/nodes/guardrail_node.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/nodes/guardrail_node.py)**: Implements the node logic, orchestrates the LLM call, parses `GuardrailScoring`, and applies the threshold comparison.
- **[`src/services/agents/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py)**: Contains the guardrail validation prompt template that steers the LLM toward evaluating relevance and safety.
- **[`src/services/agents/models.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py)**: Defines the `GuardrailScoring` Pydantic schema that structures the LLM's output into machine-readable scores and explanations.
- **[`src/services/agents/agentic_rag.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/agentic_rag.py)**: Configures the guardrail threshold and wires the node into the overall RAG graph workflow.

## Summary

- The Guardrail node uses **LLM-driven relevance scoring** to detect out-of-domain content in the `jamwithai/production-agentic-rag-course` pipeline.
- Detection relies on a **validation prompt** ([`src/services/agents/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py)) that scores responses for relevance and safety.
- Output is structured via the **`GuardrailScoring`** model ([`src/services/agents/models.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py)), enforcing a 0-100 integer score and explanatory reason.
- The node compares scores against **`guardrail_threshold`** from the graph configuration to determine Pass/Fail status.
- Failed checks trigger pipeline abort or fallback responses, protecting the system from off-topic or unsafe outputs.

## Frequently Asked Questions

### What triggers an out-of-domain failure in the Guardrail node?

An out-of-domain failure occurs when the LLM returns a `GuardrailScoring` result with a `score` below the configured `guardrail_threshold` (typically set around 30-50). This indicates the model judged the answer irrelevant to AI research papers or detected unsafe content, prompting the node to mark the output as Fail and halt downstream processing.

### How does the GuardrailScoring model structure the LLM output?

According to [`src/services/agents/models.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/models.py), the `GuardrailScoring` Pydantic model requires the LLM to return two fields: a `score` integer between 0 and 100 measuring domain relevance, and a `reason` string explaining the evaluation. The `with_structured_output(GuardrailScoring)` method enforces this schema at the API level, ensuring type-safe parsing.

### Where is the guardrail threshold configured?

The threshold is defined in the graph configuration accessible via `self.graph_config.guardrail_threshold` within the Guardrail node. This value is typically initialized in [`src/services/agents/agentic_rag.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/agentic_rag.py) where the RAG graph is constructed and node parameters are injected, allowing per-deployment customization of sensitivity.

### Can the validation criteria be customized?

Yes. The evaluation criteria are defined in the guardrail validation prompt located in [`src/services/agents/prompts.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/prompts.py). Developers can modify this prompt template to adjust the domain definition (e.g., changing from AI papers to medical research) or refine safety guidelines, though the core scoring mechanism via `GuardrailScoring` remains unchanged.