# How Needle 2 Enforces Output Conformity to Declared Schemas

> Learn how Needle 2 enforces output conformity to declared schemas. Discover its three-stage grounding and validation pipeline for Pydantic models and structured payloads.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-11

---

**Needle 2 guarantees that model-generated outputs strictly adhere to JSON schemas by normalizing Pydantic models into JSON-Schema dictionaries, generating structured payloads, and recursively validating every field against source text through a three-stage grounding and validation pipeline.**

Needle 2 is an open-source structured extraction framework that ensures language model outputs conform precisely to declared schemas. Unlike approaches relying on post-hoc heuristics or gentle prompting, the library implements a rigorous validation pipeline that statically walks JSON schemas and cross-checks concrete values against input text before exposing results to users.

## Schema Normalization: Converting Python Types to JSON-Schema

Needle 2 begins enforcement by transforming Python callables and Pydantic models into standards-compliant JSON-Schema dictionaries. In [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py), the `build_schema` and `pydantic_schema` functions handle this conversion, extracting field definitions, descriptions, and format constraints from type annotations.

The resulting schema objects are attached directly to decorated functions via `fn._needle_tool` or accumulated in the engine's internal `_tool_schemas` list (lines 173–214 of [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)). This attachment ensures that every tool or extraction request carries its structural contract through the execution pipeline.

## Generation and Initial Extraction

When processing requests, the public method `Needle.extract(text, schema, …)` delegates to an internal helper `extract` defined in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py). This stage leverages [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) as a low-level inference wrapper to prompt the underlying language model for a JSON payload matching the requested schema shape. However, Needle 2 treats this initial output as **untrusted raw material** rather than a validated result.

The generation phase produces structured data that superficially resembles the target schema, but the framework deliberately suspends judgment until the subsequent validation stage completes its verification pass.

## Grounding and Validation Pipeline

The core enforcement mechanism occurs in the grounding and validation stage, which recursively verifies that generated values actually exist in or are supported by the source text.

### Temporal Source Analysis

Before validating structure, Needle 2 extracts temporal context from the input. The `_source_years` and `_licensed_years` methods (lines 78–99 of [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)) scan the original input text and optional system prompts to identify all calendar years explicitly mentioned in the source material. These years form the "licensed" temporal bounds that constrain date and date-time fields in the output.

### Recursive Schema Walking

The `_walk_grounding` function implements the primary validation logic. This recursive walker accepts the JSON schema, the generated arguments object, and the set of licensed years, then traverses the data structure according to schema definitions:

- It resolves `$ref` pointers to handle complex nested schemas.
- For string fields with `date` or `date-time` formats, it verifies that the year component appears in the licensed years set extracted from the source.
- For objects and arrays, it follows `properties`, `items`, and `anyOf`/`oneOf` branches, ignoring nullable branches when encountering null values.
- It records every successfully verified path in a `checked` collection and logs failures for later analysis.

### Final Validation and Error Handling

The `_validate_extraction` method (lines 68–80 of [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)) combines grounding results with any previously recorded ungrounded paths, applies negation checks, and raises an `ExtractionValidationError` if unresolved failures remain. This exception includes a precise list of schema paths that failed grounding, enabling programmatic retry logic or user-facing error messages that specify exactly which fields violated the schema constraints.

If validation succeeds, the payload returns to the caller with guaranteed structural and semantic conformity to the declared schema.

## Implementation Example: Validating Weather Data Extraction

The following demonstration shows schema enforcement in practice, where an invalid date would trigger validation failure:

```python
from needle import Needle, ExtractionValidationError
from pydantic import BaseModel, Field

# Declare schema using Pydantic

class WeatherRequest(BaseModel):
    """Weather query."""
    city: str = Field(..., description="City name")
    date: str = Field(..., format="date", description="Date of interest")

# Initialize engine

engine = Needle()

# Attempt extraction with grounding verification

prompt = "What will the weather be in Paris on 2024-07-15?"
try:
    result = engine.extract(prompt, WeatherRequest)
    print(result)  # → {'city': 'Paris', 'date': '2024-07-15'}

except ExtractionValidationError as err:
    print("Validation failed:", err)

```

*If the model hallucinated "2025-07-15" (a year not present in the prompt), the engine raises `ExtractionValidationError` because 2025 is not grounded in the source text.*

## Summary

- **Needle 2 enforces output conformity** through a three-stage pipeline: schema normalization, structured generation, and recursive grounding validation.
- **Schema definitions** are built by `build_schema` and `pydantic_schema` in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py), then attached to tools or stored in `_tool_schemas`.
- **Temporal grounding** ensures date and date-time fields reference years actually present in source text via `_source_years` and `_licensed_years`.
- **Recursive validation** via `_walk_grounding` checks every schema path, resolves references, and tracks failures through `checked` and `failures` collections.
- **Strict error handling** through `_validate_extraction` raises `ExtractionValidationError` with specific path information when grounding fails.

## Frequently Asked Questions

### How does Needle 2 handle complex nested schemas with references?

Needle 2 processes nested structures through the `_walk_grounding` function, which actively resolves `$ref` pointers while traversing object properties and array items. The walker handles `anyOf` and `oneOf` branches by exploring valid alternatives and ignores nullable branches when encountering null values, ensuring comprehensive validation across arbitrarily deep schema hierarchies defined in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py).

### What happens when validation fails during extraction?

When validation fails, Needle 2 raises an `ExtractionValidationError` that contains a detailed list of schema paths that could not be grounded in the source text. This exception allows calling code to identify exactly which fields violated constraints—such as dates with unsupported years or missing required properties—and implement retry logic with adjusted prompts or fallbacks.

### Can Needle 2 validate schemas without temporal grounding?

While Needle 2 specializes in temporal grounding for date and date-time fields, the validation framework is schema-agnostic and checks all field types against their JSON-Schema definitions. The `_walk_grounding` mechanism verifies structural conformity for objects, arrays, and primitive types regardless of whether temporal constraints apply, though the library is optimized for use cases requiring source-text verification.

### Where does schema normalization occur in the codebase?

Schema normalization occurs in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) through the `build_schema` and `pydantic_schema` functions, which convert Python callables and Pydantic models into JSON-Schema dictionaries. These normalized schemas are then stored in the engine's `_tool_schemas` list or attached directly to function objects via the `_needle_tool` attribute, as implemented in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) between lines 173 and 214.