Constrained Generation: Beyond Hope-Based Parsing

Stop Hoping for Valid JSON

The default way most people get structured output from an LLM is: ask for JSON, get something back, try to parse it, catch the exception, maybe retry with a sterner prompt. That whole loop exists because the model is free, at every step, to emit whatever token it wants. It just usually cooperates. Constrained generation removes the "usually." Instead of asking nicely and validating afterward, you make the invalid output impossible to produce in the first place.

The Core Mechanism: Masking Invalid Tokens

An LLM generates text one token at a time. At each step it computes a probability distribution over its entire vocabulary — tens of thousands of possible next tokens — and samples from it. Constrained generation sits between that distribution and the sampler. Given a grammar (a JSON schema, a regex, a formal context-free grammar), the engine tracks which tokens are even syntactically valid at the current position, and sets the logits for every invalid token to negative infinity before sampling happens. The model doesn't choose not to emit an invalid token. It literally cannot — the probability of sampling it is zero, every time, at every step.

That's a different category of guarantee than prompting. A well-crafted prompt raises the odds of good output. A grammar mask makes bad output structurally unreachable, the same way a database schema constraint doesn't just discourage a bad insert, it rejects it.

flowchart TD
    Step["Decode step: compute logits
over full vocabulary"] --> Grammar["Grammar engine: which tokens
are valid at this position?"]
    Grammar --> Mask["Mask invalid tokens to -inf"]
    Mask --> Sample["Sample from remaining
valid tokens only"]
    Sample --> Emit["Emit token, advance grammar state"]
    Emit -->|Not done| Step
    Emit -->|Structure complete| Done["Guaranteed-valid output"]

    style Mask fill:#dc2626,color:#fff,stroke:#b91c1c
    style Sample fill:#059669,color:#fff,stroke:#047857
    style Done fill:#4f46e5,color:#fff,stroke:#4338ca
      

Three Tiers, Not Two

It's easy to lump "JSON mode" and "Outlines/llama.cpp grammars" together because they both promise valid structured output. Mechanically they're close cousins — both are token-level constraint under the hood — but where the constraint runs is the whole difference in practice.

The reason this matters: provider structured output only exists for the providers that built it, only for the formats they chose to support (usually JSON Schema, not arbitrary regex or a custom DSL), and only when you're calling their hosted API. Run Llama or Mistral locally, or need to constrain output to a grammar that isn't JSON at all — a SQL dialect, a config file format, a domain-specific language — and you need the self-run tier. That's where Outlines, Guidance, LMQL, and llama.cpp's GBNF grammars live, and it's currently the most reliable way to get structured output out of an open-weight model.

A Concrete Grammar Example

A llama.cpp GBNF grammar constraining output to a fixed JSON shape — a support ticket classification, nothing the model can wander outside of:

root ::= "{" ws "\"category\":" ws category "," ws "\"priority\":" ws priority "}" ws

category ::= "\"bug\"" | "\"feature-request\"" | "\"support\""
priority ::= "\"low\"" | "\"medium\"" | "\"high\""
ws ::= [ \t\n]*

Passed to llama.cpp, this grammar makes it impossible for the model to emit a fifth category or a priority outside the three allowed values — not unlikely, impossible. The same constraint expressed with Outlines, against an open-weight model served locally:

import outlines
from pydantic import BaseModel
from typing import Literal

class TicketClassification(BaseModel):
    category: Literal["bug", "feature-request", "support"]
    priority: Literal["low", "medium", "high"]

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.3")
generator = outlines.generate.json(model, TicketClassification)

result = generator("The checkout button doesn't work on mobile")
# result is guaranteed to be a valid TicketClassification instance —
# no try/except around a JSON.parse, because invalid JSON was never reachable.

The Trade-off: Grammar Tunnel Vision

Constrained decoding isn't free. The failure mode people run into after the initial "this never breaks" honeymoon is what practitioners sometimes call grammar tunnel vision: forcing the model down a rigid token path early in generation can hurt output quality, especially for anything that benefits from reasoning before committing to an answer. If your grammar forces {"answer": as the very first tokens, the model never gets to think out loud first — and a lot of what makes chain-of-thought prompting effective is exactly that room to reason before answering. Force the structure too early and you can get a confidently well-formed wrong answer instead of a correctly-reasoned one wrapped in occasionally-messy JSON. The fix, when it matters, is to let the model reason in an unconstrained scratch field first and constrain only the final structured field — not to abandon constraints altogether.

When to Reach for It

Treat this as an escalation ladder, not a first resort. Start with a provider's native structured output if you're on a hosted frontier model and a JSON schema covers what you need — it's the least code, and for most JSON-shaped extraction and classification tasks it's genuinely enough. If that's unavailable, or the failure rate on validation is nonzero and costing you retries, move to validate-and-repair: parse, and on failure, send the error back to the model for a self-correction pass rather than a blind retry. Reach for full constrained decoding — Outlines, llama.cpp grammars — when you're running an open-weight model without hosted structured output, or when the format isn't JSON at all and a schema-based tool won't express it. It's the strictest guarantee and the most implementation complexity, which is exactly why it shouldn't be the default reach for every structured-output problem.

Pitfalls

Constraining too early in the generation

Forcing structure from token one removes the model's ability to reason before committing. For anything non-trivial, generate a reasoning trace in an unconstrained field first, then constrain only the final answer field.

Assuming provider JSON mode and self-run grammars are the same guarantee

Not every provider implements structured output as strict token masking under the hood — some layer validation and retry behind a JSON-mode-shaped API. Check what your specific provider actually guarantees before treating "JSON mode" as equivalent to a grammar you compiled yourself.

Writing an overly permissive grammar

A grammar with too many optional fields or loose regex classes barely constrains anything, and you lose most of the reliability benefit while paying the engineering cost of maintaining a grammar file. Write it as tight as the actual schema allows.

Ignoring the performance cost

Grammar-constrained decoding adds per-token overhead to compute and check the valid-token mask, and complex grammars with deep nesting can slow generation noticeably compared to unconstrained decoding. Benchmark it on your actual grammar and hardware before assuming the overhead is negligible.

Key Takeaways