# How to Use the `extract()` Method for Structured Data Extraction in Needle

> Learn how to use Needle's extract() method for structured data extraction. Effortlessly pull information from unstructured text using a schema-driven approach.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-26

---

**The `extract()` method provides a one-shot, schema-driven approach to pull structured information from unstructured text by creating a temporary agent with a grammar-constrained single tool.**

Needle's `extract()` function, available as both a module-level helper and a class method on the `Needle` agent, eliminates the need for manual prompt engineering when you need to parse free-form text into typed data structures. This guide covers the implementation details from the cactus-compute/needle source code and practical usage patterns.

## Overview of the `extract()` Method

The `extract()` method solves a common problem in LLM applications: reliably converting unstructured text into structured, validated data. Unlike general prompting approaches, Needle's implementation guarantees schema conformance through grammar-constrained generation.

According to the Needle source code, the method works by:

1. Creating a temporary `Needle` agent with your schema as the **only** available tool
2. Re-initializing the shared inference engine with this single-tool context
3. Running one completion pass where the model's output grammar is constrained to exactly one function call
4. Returning the parsed arguments as either a Pydantic model instance or plain dictionary

This architecture ensures the model cannot emit malformed JSON or hallucinate extra fields.

## How `extract()` Works Under the Hood

### Module-Level Function vs. Class Method

Needle provides two equivalent entry points. In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the `Needle.extract()` class method (lines 49-52) is a thin wrapper that forwards to the module-level `extract()` helper:

```python

# From needle/__init__.py lines 49-52

def extract(self, text, schema, system=None):
    """Extract structured data from text using this agent's weights."""
    return extract(text, schema, system=system, weights=self.weights)

```

The core implementation lives in the module-level `extract()` function (lines 66-79). This function:

1. **Instantiates a temporary agent** with the schema as its sole tool (lines 66-73)
2. **Calls `agent.complete()`** to generate the structured response (line 74)
3. **Extracts the first function call's arguments** from the response (lines 74-78)
4. **Returns a Pydantic model or dict** based on the input schema type (lines 78-79)

### Grammar-Constrained Generation

The key to reliable extraction is Needle's use of constrained decoding. When only one tool is declared, the grammar admits exactly one function call. This means:

- The model **cannot** output invalid JSON
- The model **cannot** omit required fields
- The model **cannot** add unexpected fields

Confidence gating still applies: the extraction only succeeds if the model's confidence exceeds your configured threshold.

## Using `extract()` with Pydantic Models

The recommended approach uses **Pydantic models** for automatic validation and IDE support.

```python
from pydantic import BaseModel
import needle

# Define your extraction schema

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

# Extract structured data in one call

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)

print(invoice.vendor)    # → Acme Corp

print(invoice.total)     # → 1200.0

print(invoice.due_date)  # → 2026-09-01

```

Pydantic models provide:
- **Type validation** — invalid values raise clear errors
- **IDE autocomplete** — full IntelliSense on extracted fields
- **Nested structures** — complex schemas with sub-models

## Using `extract()` with Raw JSON Schema

For dynamic schemas or when Pydantic isn't appropriate, pass a raw JSON Schema dictionary:

```python
import needle

schema = {
    "name": "receipt",
    "description": "Purchase receipt",
    "parameters": {
        "type": "object",
        "properties": {
            "merchant": {"type": "string"},
            "total": {"type": "number"},
            "currency": {"type": "string"},
        },
        "required": ["merchant", "total"],
    },
}

receipt = needle.extract(
    "GreenMart receipt: oat milk 3.50, total 7.75 paid by visa",
    schema,
)

print(receipt)  # → {'merchant': 'GreenMart', 'total': 7.75, 'currency': 'USD'}

```

Note that raw schemas return plain Python dictionaries, not typed objects.

## Using `extract()` on an Existing Agent Instance

When you already have a `Needle` agent with loaded weights, use the instance method to avoid re-initializing the inference engine:

```python
import needle

# Reuse existing agent with your weights

agent = needle.Needle()

person = agent.extract(
    "Dr. Sarah Chen, 42, joined as CTO in March 2023",
    {
        "name": "executive",
        "parameters": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "age": {"type": "integer"},
                "title": {"type": "string"},
                "start_date": {"type": "string"},
            },
            "required": ["name", "title"],
        },
    },
)

print(person["name"])   # → Sarah Chen

print(person["age"])    # → 42

```

This approach respects the agent's existing `weights` configuration while temporarily overriding its toolset.

## Key Parameters and Options

| Parameter | Type | Description |
|-----------|------|-------------|
| `text` | `str` | The unstructured source text to parse |
| `schema` | `BaseModel` or `dict` | Pydantic model class or JSON Schema dict defining the output structure |
| `system` | `str` or `None` | Optional system prompt to guide extraction behavior |

The `system` parameter allows light customization without breaking the grammar constraints. For example, pass instructions about date formats or inferring missing fields.

## Performance and Reliability Considerations

- **Single-pass extraction** — No iterative refinement loops, minimizing token usage
- **Deterministic grammar** — Same input always produces valid, parseable output
- **Confidence thresholding** — Low-confidence extractions can be flagged for review
- **Lightweight temporaries** — The temporary agent is garbage-collected after extraction

## Summary

- **`extract()`** provides guaranteed schema-conformant extraction through grammar-constrained generation
- **Two entry points**: `needle.extract()` for standalone use, `agent.extract()` for existing instances
- **Schema flexibility**: Accepts Pydantic models (returns instances) or raw dicts (returns dicts)
- **No prompt engineering required**: Declare your target shape and let Needle handle the rest
- **Implementation locations**: [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) lines 49-52 (wrapper) and 66-79 (core logic)

## Frequently Asked Questions

### What happens if the model's confidence is below the threshold?

Needle's confidence gating applies to `extract()` just like other completions. If the model's confidence falls below your configured threshold, the extraction will fail or return `None` depending on your error-handling settings. This prevents low-quality extractions from propagating through your system.

### Can I use nested Pydantic models with `extract()`?

Yes. Needle compiles the full JSON Schema from your Pydantic model, including nested structures. Define sub-models as separate `BaseModel` classes and reference them as field types in your main extraction schema. The grammar constraint handles arbitrarily complex nesting.

### Why does `extract()` create a temporary agent instead of using my existing agent's tools?

The temporary agent ensures **single-tool, single-call semantics**. If your main agent had multiple tools available, the grammar would allow multiple function calls, breaking the extraction guarantee. By isolating to one tool, Needle constrains the output to exactly one structured object.

### Is there a performance penalty for creating temporary agents?

Minimal. The temporary agent shares the underlying inference engine and weights with your main agent. The only overhead is re-initializing the tool context, which is negligible compared to the generation cost. For bulk extractions, consider batching or reusing the module-level `needle.extract()` function.