# How the Byte-Level Grammar Ensures Schema-Conformant JSON Output in Needle

> Needle's byte-level grammar ensures 100% schema-conformant JSON output by validating tokens during generation. Generate valid JSON without post-processing.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-08-23

---

**Needle uses a byte-level grammar compiled from JSON schemas to validate every token during generation, guaranteeing 100% schema-conformant JSON output without post-processing or repair.**

The Needle library solves the "JSON mode" reliability problem by moving schema enforcement from post-generation parsing into the **decoding process itself**. Instead of allowing a language model to produce arbitrary tokens and then hoping the result parses correctly, Needle compiles your schemas into a deterministic finite automaton that operates at the byte level. This article explains how this mechanism works, from Python function definition to guaranteed-valid output.

## Building the Schema from Python Definitions

Every JSON tool call starts with a Python function or Pydantic model. When you decorate a function with `@needle.tool` or pass a model to `needle.extract`, the library introspects the signature to construct a complete JSON-Schema representation.

The heavy lifting happens in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py). The `build_schema` function walks the Python signature, maps types to JSON Schema types via `_json_type`, extracts descriptions from docstrings, and handles `Field` annotations for constraints like enums, defaults, and formats.

```python

# From needle/agent/tools.py - schema construction pipeline

def build_schema(func):
    # Inspects signature, builds properties dict

    # Handles required fields, defaults, type mappings

    # Returns complete JSON-Schema dict ready for compilation

```

This schema captures everything: required parameters, optional fields with defaults, nested objects, array types, and validation constraints. The resulting dictionary is passed to the native inference engine for **grammar compilation**.

## Grammar Compilation to Byte-Level DFA

Here's where the Needle architecture differs from typical "JSON mode" implementations. Rather than using prompt engineering or post-hoc validation, Needle compiles the schema into a **byte-level grammar** that drives the decoder directly.

According to the project's README, *"a byte‑level grammar compiled from your schemas constrains every token"* (lines 10-12). This grammar is implemented as a deterministic finite automaton where:

- **Each state** represents a valid position in a JSON document conforming to the schema
- **Transitions** are valid bytes (not tokens) that keep the document in a schema-valid state
- **Terminal states** correspond to complete, valid JSON objects

The compilation happens inside the native inference engine—a 14 MB binary fetched from Hugging Face. This engine translates JSON Schema constructs like `properties`, `required`, `enum`, `type`, and `format` into corresponding DFA states and transitions.

### Why Byte-Level Matters

Operating at the **byte level** rather than token level provides finer-grained control. A single token might contain multiple JSON structural characters (e.g., `},{"key":`), or a partial match. By validating bytes, the grammar can accept or reject tokens at precisely the right boundary—preventing a token that starts valid but ends invalid from being generated at all.

## Runtime Enforcement During Decoding

During inference, the model proposes the next token. Before that token becomes part of the output, the engine simulates its byte sequence through the DFA:

1. **Propose** — The model generates token candidates with associated probabilities
2. **Validate** — Each candidate's byte representation is checked against current DFA states
3. **Filter** — Only tokens whose complete byte sequence leads to valid DFA states are retained
4. **Emit** — The highest-probability valid token is appended to the output, and DFA state updates

This loop repeats until reaching a terminal state. The result: **every generated byte has been pre-validated against the schema**.

```python

# Example: Tool definition with schema enforcement

import needle

@needle.tool
def set_lights(room: str, brightness: int = 50):
    """Set a room's lighting level."""
    return {"room": room, "brightness": brightness}

agent = needle.Needle(tools=[set_lights])
result = agent.run("Dim the kitchen lights to 10%")

# Guaranteed: result["results"] contains valid JSON matching the schema

# {"name":"set_lights","arguments":{"room":"kitchen","brightness":10}}

```

Even with complex nested structures, the grammar maintains state across the entire document. The model cannot emit a missing required field, swap types, or malform JSON punctuation—the byte-level constraints make such outputs **syntactically unreachable**.

## Supported Schema Constraints

The byte-level grammar handles the full JSON Schema vocabulary used by Needle:

| Constraint | Byte-Level Enforcement |
|------------|----------------------|
| `type: object/array/string/number/integer/boolean/null` | DFA transitions restricted to valid literals |
| `required` | State machine enforces mandatory key presence |
| `properties` / `additionalProperties` | Known keys follow defined sub-grammars; unknown keys rejected or validated per `additionalProperties` |
| `enum` | Transitions limited to exact byte sequences of allowed values |
| `const` | Single valid continuation path |
| `format` (email, date-time, uri, etc.) | Additional DFA layer for pattern validation |
| `minLength` / `maxLength` | Counter states track string/array bounds |
| `minimum` / `maximum` / `exclusiveMinimum` / `exclusiveMaximum` | Numeric range validation in state |
| `oneOf` / `anyOf` / `allOf` | Non-deterministic or product DFA constructions |

These constraints compose naturally. A nested object with enum fields, string length limits, and required properties becomes a single integrated DFA rather than separate validation layers.

## Extraction Mode: Pydantic to Guaranteed Objects

The same byte-level grammar powers `needle.extract` for structured data extraction from unstructured text:

```python
from pydantic import BaseModel
import needle

class Weather(BaseModel):
    """Weather query."""
    city: str
    units: str = "celsius"

# Schema compiled to grammar; generation constrained to match Weather

weather = needle.extract("What's the weather in Paris?", Weather)

# weather is a validated Weather instance, not raw JSON

```

Behind this method, the `pydantic_schema` helper in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) converts the Pydantic model to JSON Schema, which the engine then compiles. The returned object is instantiated directly from schema-valid bytes—no `json.loads()` followed by manual validation.

## Configuration and Escape Hatches

The library provides visibility and control over grammar-constrained decoding. In [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py), the `--no-grammar` flag disables constraint enforcement entirely, useful for debugging or comparison purposes. By default, grammar mode is enabled.

The public API surface in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (`tool`, `extract`, `Needle`) abstracts these details while ensuring schemas propagate correctly to the engine.

## Comparison with Alternative Approaches

| Approach | Failure Mode | Needle's Solution |
|----------|-----------|-----------------|
| Prompt-based JSON instructions | Model ignores format, produces invalid JSON | Byte-level grammar makes invalid outputs unreachable |
| Post-generation parsing + retry | Latency from retries; exponential backoff on failures | Zero retry loops; single-pass guaranteed valid |
| Logit bias / token banning | Coarse-grained; cannot express structural constraints | Fine-grained byte validation captures full schema |
| Grammar sampling (CFG at token level) | Token boundary issues, degraded efficiency | Byte-level DFA eliminates boundary problems |

## Summary

- **Schema construction** in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) (`build_schema`, `pydantic_schema`) converts Python definitions to JSON Schema
- **Grammar compilation** transforms schemas into byte-level DFAs inside the native inference engine
- **Runtime enforcement** validates every proposed token's bytes against the DFA before emission
- **Guaranteed output** means no parsing failures, no type mismatches, no missing required fields—every generated JSON object exactly matches its declared schema

## Frequently Asked Questions

### How does byte-level grammar differ from token-level grammar constraints?

Token-level constraints operate on vocabulary items, which may contain multiple or partial JSON structural elements. This creates boundary problems where a token is partially valid. Byte-level grammar eliminates this by validating individual bytes, allowing precise control over exactly where valid JSON structures begin and end.

### Can the grammar handle recursive or deeply nested schemas?

Yes. The DFA construction supports arbitrary nesting depth through state composition. Recursive schemas reference previously constructed sub-grammars, and the engine manages state stack depth during decoding. The README's guarantee applies regardless of nesting complexity.

### What happens if the model cannot produce any valid continuation?

The engine returns the highest-probability valid prefix completed to a valid terminal state, or signals generation failure. Because constraints are applied throughout, the model never "paints itself into a corner" with an invalid intermediate state—at every step, some valid continuation exists if the schema permits optional or default values.

### Is there performance overhead for complex schemas?

Grammar compilation occurs once per schema initialization, not per generation. The runtime overhead per token is a DFA transition lookup—O(1) with respect to schema complexity. The 14 MB native engine is optimized for this constraint checking, typically adding less than 10% latency compared to unconstrained generation for practical schemas.