Prompt EngineeringProAdvanced

Constrained Generation: Beyond Hope-Based Parsing

Enforce output structure at the token level with grammars and token masks — how it differs from native JSON mode, and when the strictness costs you reasoning quality.

By Amit UpadhyayPublished 05 JulUpdated 05 Aug 202613 minArticle 4 of 12 in Prompt Engineering

Constrained generation masks invalid tokens at every decoding step using a grammar, so output always parses; it sits above JSON mode and strict schemas, but too tight a grammar can hurt reasoning quality.

On this page

The idea at a glance

The problem

When

Code downstream needs model output in an exact shape, every time

The obvious move

Asking for the format, then parsing and retrying when it comes back wrong

Fails because

The model is free to produce any token at every step, so “usually valid” still means a steady stream of broken records at volume

The fix

Constrained generation works by

Constraining decoding so only tokens that fit the schema or grammar can be produced

At the cost of

The model can be forced to fill fields it has no information for, and complex grammars slow decoding

The scenario every mode follows

“Extract vendor, invoice number, total, currency, due date and line items from this invoice text.”

An accounts-payable pipeline extracting fields from supplier invoices
50,000 invoices a day
gpt-6-luna

Where the obvious approach breaks

0.5% bad parses × 50,000 = 250 broken records a day

If you prompt for JSON, parse it, and retry on failure.

Even a prompt that produces valid output 99.5% of the time leaves 250 invoices a day needing retries or a human. Retries add cost and latency, and some failures are valid JSON with the wrong fields, which the parser does not catch.

Helps to know firstJSON mode

Stop Hoping for Valid JSON

The default way most people get structured output from an LLM is: ask for JSON, get something back, try to parse it, catch the exception, maybe retry with a sterner prompt. That whole loop exists because the model is free, at every step, to emit whatever token it wants. It just usually cooperates. Constrained generation removes the "usually." Instead of asking nicely and validating afterward, you make the invalid output impossible to produce in the first place.

If you are still parsing and retrying

Your JSON parser has a retry loop because the model occasionally adds a trailing comma or wraps the output in prose
You're running an open-weight model locally and don't have access to a hosted structured-output feature
A downstream SQL generator occasionally produces syntactically invalid SQL that only fails at execution time
You've tuned the prompt five different ways and the failure rate on malformed output still isn't zero

Pro deep dive13 min full read

Keep reading with Pro

This is one of our advanced production deep dives. Pro unlocks the rest of it, every other deep dive, and the See, Animate, Test and Build modes on every topic.

What's inside