# Needle Grammar‑Constrained Decoding for Tool Calls: How It Prevents JSON Hallucinations

> Needle grammar-constrained decoding prevents malformed tool calls by compiling Python function schemas into byte-level grammars that constrain token generation at inference time. Avoid JSON hallucinations.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-15

---

**Needle prevents malformed tool calls by compiling Python function schemas into byte‑level grammars that constrain token generation at inference time.**

The [Needle](https://github.com/cactus-compute/needle) library implements grammar‑constrained decoding to guarantee that LLM outputs are always valid, schema‑compliant JSON. This architecture makes Needle uniquely suited for reliable tool calling on resource‑constrained edge devices.

## What Is Grammar‑Constrained Decoding?

**Grammar‑constrained decoding** restricts the model's output tokens to sequences that can only produce syntactically valid structures. In Needle, this means the model physically cannot emit malformed JSON, missing required fields, or out‑of‑range values.

The mechanism works in three stages:

- **Schema extraction** – Python type hints and `Field` constraints become JSON Schema
- **Grammar compilation** – Schemas transform into byte‑level production rules
- **Constrained generation** – Every token is validated against the grammar before acceptance

## Tool Schema Discovery in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py)

Needle's schema builder inspects decorated functions and Pydantic models to extract complete type information.

In [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) (lines 10‑41), the builder processes:

- Function name → operation identifier
- Type hints → JSON Schema `type` declarations
- `Annotated` + `needle.Field` → `minimum`, `maximum`, `enum`, and other constraints
- Docstrings → parameter descriptions

```python
from typing import Annotated
import needle

@needle.tool
def set_lights(
    room: str,
    on: bool,
    brightness: Annotated[int, needle.Field(ge=0, le=100)] = 100,
):
    """Turn a room's lights on/off and set brightness."""
    return {"room": room, "on": on, "brightness": brightness}

```

The `ge=0` and `le=100` constraints become `minimum` and `maximum` in the generated schema. The model will never output `brightness: 150` because such a token sequence violates the compiled grammar.

## Engine Initialization and Grammar Compilation

When you instantiate `needle.Needle`, schemas flow into the native inference engine.

In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 53‑66), the `Needle` class:

1. Collects all tool schemas into a list
2. Passes them to `libneedle.needle_init()` via FFI
3. Receives a compiled grammar handle used for all subsequent calls

```python
agent = needle.Needle(tools=[set_lights])  # Grammar compiled here

```

The C library (`libneedle.so`) constructs a **byte‑level grammar** where each production rule corresponds to valid JSON serialization paths. This grammar is cached and reused across calls.

## Grammar‑Aware Generation in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py)

Actual token generation enforces the grammar at every step.

In [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) (lines 26‑32), `generate_cached` creates a `DecodeCfg` from model configuration. During generation:

- `forward_cached` runs the JIT‑compiled model forward pass
- Each candidate token is submitted to the engine
- The engine checks whether appending that byte would still satisfy the grammar
- Invalid tokens are masked; only grammar‑compliant tokens are sampled

This validation happens **per‑token**, not post‑hoc. The model cannot "accidentally" produce invalid output because invalid token sequences are physically excluded from the probability distribution.

## Complete Tool‑Calling Flow

```text
Python function  →  build_schema()  →  JSON Schema
                                          ↓
Needle.__init__  →  libneedle.needle_init()  →  Compiled grammar
                                          ↓
Needle.run()     →  libneedle.needle_complete()  →  Grammar‑constrained decoding
                                          ↓
                                    Valid JSON envelope (guaranteed)

```

The final output is a JSON object that:
- Contains no syntax errors
- Includes all `required` fields
- Respects all `minimum`, `maximum`, `pattern`, and `enum` constraints
- Matches the exact structure declared in Python

## One‑Shot Structured Extraction

The same grammar mechanism powers `needle.extract()` for pure data extraction without tool execution.

```python
from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)  # → Acme Corp 1200.0

```

Here the Pydantic model's schema compiles to a grammar just like tool functions. The decoder emits only JSON matching `Invoice`'s structure.

## Why Grammar‑Constrained Decoding Matters for Edge Deployment

Needle's grammar constraint system enables reliable tool calling with **~28 MiB of RAM**. Because output validity is enforced at the token level:

- No retry loops for malformed JSON
- No separate validation step
- No prompt engineering to "beg" for correct formatting

The guarantee is structural: if the grammar compiled, valid output is provably possible and the model will only produce valid instances.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) | Schema extraction from functions and Pydantic models |
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | `Needle` class, engine initialization, agent loop |
| [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) | Decoding configuration and constrained generation |
| `libneedle.so` | Native engine with byte‑level grammar compiler |

## Summary

- Needle's **grammar‑constrained decoding** guarantees valid JSON output by compiling schemas into enforceable production rules
- The **schema builder** in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) extracts constraints from Python type hints and `Field` annotations
- **Engine initialization** in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) compiles schemas into byte‑level grammars via `libneedle.needle_init()`
- **Token‑level validation** in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) masks invalid tokens during `generate_cached`
- This architecture enables reliable tool calling on devices with minimal memory without post‑processing or retries

## Frequently Asked Questions

### How does Needle prevent the model from hallucinating extra fields?

The compiled grammar only permits token sequences that produce properties declared in the schema. Undeclared field names have no valid production rules, so the model cannot generate them. This is enforced in `libneedle.so` during every sampling step.

### Can Needle handle nested Pydantic models with optional fields?

Yes. The schema builder recursively processes nested models, and the grammar correctly handles `Optional[T]`, `Union` types, and default values. Optional fields become nullable schema properties with corresponding grammar branches.

### What happens if the model cannot satisfy the grammar constraints?

If no valid completion exists (e.g., required field missing from context), the engine returns a partial or empty result rather than invalid JSON. The `generate_cached` routine handles this gracefully; applications can detect incomplete parses and adjust prompts accordingly.

### Is grammar compilation a one‑time cost per schema set?

Yes. `needle_init` compiles grammars once during `Needle` construction. Subsequent calls to `run()` or `complete()` reuse the compiled representation. This amortization keeps per‑request latency minimal even with complex multi‑tool schemas.