Constrained Generation: Beyond Hope-Based ParsingMixed sources5 sources, drawn from a mix of official docs, project repositories and research.This rates where the sources come from. It doesn't prove every claim.
Enforce output structure at the token level with grammars and token masks — how it differs from native JSON mode, and when the strictness costs you reasoning quality.
By Amit UpadhyayPublished Updated 13 minArticle 5 of 13 in Prompt Engineering
AI assistant · answers from this article
1 of 6
Low-stakes, exploratory use
Nothing to add yet
While nothing downstream depends on every record, asking for the format and parsing the reply is enough.
On this page
Constrained generation, in one sentence
Constrained generation stops a model from producing any token that would break the required format, so invalid output cannot happen at all.
You reach for it when code downstream needs model output in an exact shape, every time. Not before: it costs the model can be forced to fill fields it has no information for, and complex grammars slow decoding.
The cheat sheet
What to use when
You do not need all of this on day one. Each row is one signal and the piece that answers it. Stop at the row your system has reached, or pick one to see it in the picture above.
Where you areThe signalWhat to useWhat it costs
None. A bad record now and then costs nothing.Nothing. Ask for JSON, and parse itNothing
The parser has a retry loop, because the model sometimes adds a trailing comma or wraps the output in prose.The provider’s structured output, with a schemaThree tiers ↓Only the formats that the provider supports
Structured output is unavailable for your case, and blind retries keep failing the same way.Validate and repair: send the error back for one correctionWhen to reach for it ↓A second call on each failure
You run an open-weight model yourself, or the output is SQL, and no hosted feature covers it.Constrained decoding that you run, with a grammarA grammar example ↓A grammar to write, and some overhead on every token
Every output is valid, and the answers have got worse.A free reasoning field first, then the constrained answerGrammar tunnel vision ↓More output tokens
The grammar has so many optional parts that almost anything gets through.A grammar as tight as the real schemaPitfalls ↓A grammar file to maintain
Start by Asking and Parsing
For exploratory work where a bad record costs nothing, ask for JSON and parse what comes back. Stronger guarantees earn their place when code downstream depends on the shape every time, and a small failure rate turns into a steady stream of broken records.
The default way most people get structured output from an LLM is: ask for JSON, get something back, try to parse it, catch the exception, maybe retry with a sterner prompt. That whole loop exists because the model is free, at every step, to emit whatever token it wants. It just usually cooperates. Constrained generation removes the "usually." Instead of asking nicely and validating afterward, you make the invalid output impossible to produce in the first place.
If you are still parsing and retrying
Your JSON parser has a retry loop because the model occasionally adds a trailing comma or wraps the output in prose
You're running an open-weight model locally and don't have access to a hosted structured-output feature
A downstream SQL generator occasionally produces syntactically invalid SQL that only fails at execution time
You've tuned the prompt five different ways and the failure rate on malformed output still isn't zero
Pro deep dive13 min full read
Keep reading with Pro
This is one of our advanced production deep dives. Pro unlocks the rest of it, every other deep dive, and the See, Animate, Test and Build modes on every topic.
What it covers
Enforce output structure at the token level with grammars and token masks — how it differs from native JSON mode, and when the strictness costs you reasoning quality.