What Is needle.extract() Used For? Structured Data Extraction in Python

needle.extract() is a convenience wrapper that performs one-shot structured extraction from free-form text using the Needle engine, returning instantiated Pydantic models or dictionaries while optionally validating temporal grounding against the source text.

Needle is an open-source structured extraction engine that uses native C++ backends to parse free-form text into typed schemas. The needle.extract() function provides a high-level API for users who need immediate, one-shot extraction without managing a persistent Needle agent instance.

How needle.extract() Works Under the Hood

The implementation lives in needle/__init__.py at lines 83-95, where the function acts as a factory for temporary extraction agents. When invoked, needle.extract() performs three core operations:

• Agent Initialization: It instantiates a temporary Needle agent, passing the supplied schema (either a Pydantic model or JSON-schema dictionary) as a tool list with tools=[schema]
• Engine Invocation: It calls agent._complete(), which forwards the prompt to the native C++ engine via needle_complete and parses the JSON envelope returned by the engine
• Resource Cleanup: The function ensures agent.close() runs in a finally block to release native resources regardless of success or failure

def extract(text: str, schema: type | dict, system: str | None = None,
            max_new_tokens: int = 256, weights: str | None = None,
            strict: bool = True) -> object:
    """One-shot structured extraction using the matching native engine."""
    selected = weights
    generation = _weight_generation(selected) if selected else 2
    _track("extract", {"n_tools": 1,
                       "tuned": bool(selected),
                       "generation": generation})
    agent = Needle(tools=[schema], system=system, weights=selected)
    try:
        response = agent._complete(text, max_new_tokens)
    finally:
        agent.close()
    # …validation and return handling omitted for brevity…

Parameters and Configuration Options

The needle.extract() signature accepts several parameters that control extraction behavior:

• text: The input string containing unstructured data to parse
• schema: Either a Pydantic BaseModel subclass or a JSON-schema dictionary defining the output structure
• system: Optional system prompt string to guide the extraction behavior
• max_new_tokens: Integer limiting the generation length (default: 256)
• weights: Path to a custom .cact fine-tuned model file; when provided, the function routes to FineTuneWorker in needle/_worker.py
• strict: Boolean flag (default: True) enabling temporal validation against years mentioned in the source text

Validation and Strict Mode

When strict=True (the default), needle.extract() invokes the _validate_extraction helper (lines 68-81 in needle/__init__.py) to perform temporal grounding checks. This validation ensures that date and year values in the extracted output actually appear in or can be directly inferred from the source text.

If the engine produces values that contradict the input or hallucinate dates not present in the text, the function raises ExtractionValidationError. Setting strict=False bypasses these checks, returning raw extraction results without temporal verification.

Practical Code Examples

Extracting with Pydantic Models

Define your schema as a Pydantic model for type-safe extraction with automatic validation:

import pydantic
from needle import extract

class Contact(pydantic.BaseModel):
    name: str
    email: str

text = "John Doe can be reached at john@doe.com"
contact = extract(text, Contact)
print(contact)  # → Contact(name='John Doe', email='john@doe.com')

Using JSON Schema Dictionaries

For dynamic schemas or runtime-defined structures, pass a plain dictionary conforming to JSON Schema:

schema = {
    "type": "object",
    "properties": {
        "city": {"type": "string"},
        "temp_c": {"type": "number"}
    },
    "required": ["city", "temp_c"]
}

result = extract("Weather in Paris: 21 °C", schema)
print(result)  # → {'city': 'Paris', 'temp_c': 21}

Handling Strict Validation Errors

When strict mode detects ungrounded temporal values, it raises ExtractionValidationError:

from needle import extract, ExtractionValidationError

try:
    # Input mentions 2025, but extraction asks for year 2020 → validation error

    extract(
        "Report due on 2025-09-01",
        {"type": "object", "properties": {"year": {"type": "integer"}}}
    )
except ExtractionValidationError as e:
    print("Validation failed:", e)

Using Custom Fine-Tuned Weights

Specify a .cact weights file to use custom fine-tuned models instead of the base engine:

extracted = extract(
    "Alice bought 3 apples for $5",
    {
        "type": "object",
        "properties": {
            "buyer": {"type": "string"},
            "quantity": {"type": "integer"},
            "price": {"type": "number"}
        }
    },
    weights="fine_tuned.cact"  # Path to custom weights

)

Summary

• needle.extract() provides a stateless, one-shot API for structured data extraction without manual Needle agent management
• The function resides in needle/__init__.py and delegates to the native C++ engine via Needle._complete()
• It accepts both Pydantic models and JSON-schema dictionaries as extraction targets
• Strict mode (enabled by default) validates temporal values against source text, raising ExtractionValidationError for ungrounded results
• Support for custom .cact weight files enables fine-tuned model inference through the weights parameter

Frequently Asked Questions

What is the difference between needle.extract() and instantiating a Needle agent directly?

needle.extract() is a convenience wrapper that creates a temporary Needle agent, runs a single extraction, and automatically cleans up resources. Use it for one-off extractions where you don't need to maintain agent state across multiple calls. For batch processing or persistent configurations, instantiate Needle directly with Needle(tools=[schema]) and reuse the agent instance.

Can I use needle.extract() without Pydantic installed?

Yes. While Pydantic models provide the best developer experience with type hints and validation, needle.extract() accepts standard JSON-schema dictionaries as the schema parameter. The engine processes both formats identically, making the function usable in lightweight environments where Pydantic dependencies are undesirable.

Why does needle.extract() raise ExtractionValidationError for dates that seem correct?

When strict=True (the default), the function runs _validate_extraction to ensure temporal values like years exist in the source text. If the LLM extracts "2020" but the input only mentions "2025", the validation fails to prevent hallucination. Either verify your input text contains the temporal references you're extracting, or set strict=False to disable this grounding check.

How do I use a custom fine-tuned model with needle.extract()?

Pass the file path to your .cact weights file via the weights parameter. The function routes to FineTuneWorker in needle/_worker.py when weights are specified, enabling inference using models fine-tuned for specific extraction domains like medical records or legal documents.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}