Grammar-Based Output Constraints in llama.cpp: A Complete Guide to Structured JSON Generation

Grammar-based output constraints in llama.cpp use a GBNF (Generalized Backus-Naur Form) parser to compile formal grammars into deterministic state machines, filtering candidate tokens during sampling to guarantee syntactically valid JSON output without post-processing.

The llama.cpp repository ships with a built-in grammar engine that constrains token generation to user-defined formal languages, eliminating the need for external validation when producing structured outputs. By implementing grammar-based output constraints, developers can force large language models to emit well-formed JSON, custom domain-specific languages, or any context-free structure with complete syntactic guarantees. This mechanism compiles GBNF specifications into deterministic automata that the sampler consults on every token candidate, rejecting any token that would violate the grammar.

How Grammar-Based Constraints Work in llama.cpp

The GBNF Parser and State Machine

The core grammar engine lives in src/llama-grammar.h and src/llama-grammar.cpp. When you provide a grammar, the llama_grammar_parser::parse function tokenizes the GBNF text and builds a map of symbol IDs alongside production rules represented as llama_grammar_element structures.

The parser produces a C-style array of rules via c_rules() and identifies a root symbol ID. The llama_grammar_init_impl function then allocates the runtime llama_grammar structure, storing the compiled deterministic automaton that tracks valid parse states throughout generation.

Sampling Integration

Grammar validation integrates directly into the sampling pipeline through common/sampling.h and common/sampling.cpp. During each generation step, the sampler constructs a list of candidate token IDs (common_sampler_candidate). When accept_grammar is set to true (the default for completions, disabled for prompt processing as seen in tools/completion/completion.cpp lines 713-738), the system calls llama_grammar_accept_token(grammar, stack, token_id) for each candidate.

Only tokens that maintain a valid grammar stack state are retained. Because this filtering occurs before softmax sampling, the model physically cannot emit malformed JSON or violate structural constraints, providing guaranteed syntactic correctness without post-hoc validation.

Lazy Grammar and Triggers

For bounded outputs, llama.cpp supports lazy grammar mode via optional trigger conditions defined in common/sampling.h (common_grammar_trigger structure). When grammar_lazy is enabled, the engine stops sampling immediately upon satisfying any specified trigger rule.

The server API exposes this through "grammar_lazy" and "grammar_triggers" fields, parsed in tools/server/server-task.cpp (lines 316-403) and validated in tools/server/server-common.cpp (lines 918-1094). This feature proves essential for generating single JSON objects or specific structural fragments without unnecessary continuation tokens.

Implementing Grammar-Based Output Constraints: 4 Methods

Method 1: CLI with GBNF File

Define your grammar using GBNF syntax. For a simple JSON object:

root ::= '{' members '}'
members ::= pair (',' pair)*
pair ::= string ':' value
value ::= string | number | object | array | "true" | "false" | "null"
object ::= '{' members? '}'
array ::= '[' elements? ']'
elements ::= value (',' value)*
string ::= '"' (/[^\"]+/) '"'
number ::= /-?[0-9]+(\.[0-9]+)?([eE][+-]?[0-9]+)?/

Save this as json_object.gbnf, then run:

./build/bin/main -m models/7B/ggml-model-q4_0.bin \
    -p "Generate a user profile in JSON:" \
    --grammar json_object.gbnf \
    -n 200

The --grammar flag instructs the parser in src/llama-grammar.cpp to compile the file and enforce constraints during sampling.

Method 2: CLI with JSON Schema

Write a standard JSON Schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "name":   { "type": "string" },
    "age":    { "type": "integer" },
    "email":  { "type": "string", "format": "email" }
  },
  "required": ["name", "age"]
}

Save as user_schema.json and invoke:

./build/bin/main -m models/7B/ggml-model-q4_0.bin \
    -p "Create a JSON user record:" \
    --json-schema user_schema.json \
    -n 200

Internally, the repository calls json_schema_to_grammar() (defined in common/json-schema-to-grammar.cpp line 321) to produce the equivalent GBNF before feeding it to the sampler.

Method 3: Server HTTP API

For production deployments using the llama.cpp server, send grammar constraints via JSON payload:

curl -X POST http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama-7b",
        "prompt": "Give me a JSON invoice:",
        "max_tokens": 256,
        "grammar": "root ::= '\"' '\"' { members } '\"' '\"' ...",
        "grammar_lazy": true,
        "grammar_triggers": [{ "type": 0, "word": "}" }]
      }'

The request body fields "grammar", "grammar_lazy" and "grammar_triggers" are parsed in tools/server/server-task.cpp (lines 316-403) and validated in tools/server/server-common.cpp (lines 918-1094) before passing the compiled grammar to the sampling engine.

Method 4: Python Wrapper

The llama-cpp-python binding exposes the same functionality:

import llama_cpp

model = llama_cpp.Llama(model_path="models/7B/ggml-model-q4_0.bin")
grammar = open("json_object.gbnf").read()

output = model(
    prompt="Return a product catalog as JSON:",
    max_tokens=300,
    grammar=grammar,
    grammar_lazy=True,
    grammar_triggers=[{"type": 0, "word": "}"}]
)

print(output["choices"][0]["text"])

The Python binding forwards the grammar argument to llama_cpp_set_grammar (see the wrapper in python/llama_cpp/__init__.py), which ultimately invokes the same C++ path as the CLI and server.

Key Source Files and Architecture Reference

File	Role
src/llama-grammar.h / src/llama-grammar.cpp	Core GBNF parser, grammar compiler, and runtime state machine (llama_grammar_parser::parse, llama_grammar_init_impl).
common/json-schema-to-grammar.h / common/json-schema-to-grammar.cpp	Converts JSON Schema documents to GBNF (json_schema_to_grammar() at line 321).
common/sampling.h / common/sampling.cpp	Integrates grammar validation into the token sampler (common_sampler_accept, llama_grammar_accept_token).
tools/server/server-task.cpp	Parses HTTP API grammar fields (lines 316-403).
tools/server/server-common.cpp	Validates grammar requests (lines 918-1094).
tests/test-llama-grammar.cpp	Unit tests for grammar parsing and stack handling.
tests/test-json-schema-to-grammar.cpp	Tests for JSON Schema conversion pipeline.

Summary

• Grammar-based output constraints compile GBNF specifications into deterministic state machines that filter tokens during sampling, guaranteeing syntactically valid JSON.
• The GBNF parser (src/llama-grammar.cpp) and JSON-Schema converter (common/json-schema-to-grammar.cpp) provide flexible input methods for defining constraints.
• Integration points include the CLI (--grammar, --json-schema), HTTP server ("grammar", "grammar_lazy", "grammar_triggers"), and Python bindings (grammar= parameter).
• Lazy grammar mode with triggers allows early termination when specific structural boundaries are reached, optimizing token usage for bounded outputs.

Frequently Asked Questions

What is GBNF and why does llama.cpp use it?

GBNF (Generalized Backus-Naur Form) is a formal notation for describing context-free grammars. llama.cpp uses GBNF because it compiles efficiently into a deterministic state machine that can validate token sequences in real-time during sampling. The parser in src/llama-grammar.cpp converts GBNF rules into llama_grammar_element structures that the sampler queries for every candidate token, ensuring only syntactically valid sequences are emitted.

Can I use standard JSON Schema directly without converting to GBNF?

Yes. The repository includes a built-in converter in common/json-schema-to-grammar.cpp that translates standard JSON Schema documents into equivalent GBNF automatically. When using the CLI with --json-schema or the server API with a schema reference, the system invokes json_schema_to_grammar() (line 321) to perform the conversion before compilation, allowing you to work with familiar schema definitions while still benefiting from grammar-constrained sampling.

How does grammar-based sampling affect generation speed?

Grammar-based sampling adds minimal overhead because the state machine evaluation happens in C++ during the candidate filtering phase. The llama_grammar_accept_token function (called from common_sampler_accept in common/sampling.cpp) performs a fast stack-based validation against pre-compiled rules. While complex grammars with many recursive rules may introduce slight latency, the deterministic filtering typically adds less than 5-10% overhead compared to unconstrained generation, making it suitable for production APIs.

What happens if the model cannot satisfy the grammar constraints?

If the model reaches a state where no valid continuation tokens exist according to the grammar rules, the sampler will return an empty candidate list, effectively stopping generation. In common/sampling.cpp, when llama_grammar_accept_token rejects all candidates for a given position, the generation halts rather than emitting invalid tokens. For lazy grammar mode with triggers, the system stops early when triggers are satisfied, but if the model fails to reach a valid parse state before hitting token limits, the output may be incomplete though still syntactically valid up to the point of termination.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}