The guarantee is real — but it’s not the model being smarter#

The technique behind structured output is called constrained decoding (or grammar-constrained sampling), and the key insight is that it has nothing to do with the neural network.

Normal generation works like this: the transformer produces raw logits over the entire vocabulary — a hundred thousand tokens, each with some probability — and sampling picks one. Any token is possible, including the one that breaks your JSON.

Constrained decoding adds one step. Your JSON Schema gets compiled into a finite state machine. At every generation step, the FSM knows which tokens are legal right now — and every illegal token has its logit set to negative infinity before sampling. The model’s weights never change. What changes is the sampling space.

It is mathematically impossible to produce a token that violates the schema. Not because the model got smarter — because the illegal paths are closed.

The mask sits between the LLM’s raw logits and the sampling step — a JSON Schema compiled once into an FSM, checked at every token.

Left: the model’s raw next-token scores — "the" and "banana" outrank every legal continuation. Right: the FSM masks every schema-illegal token to −∞ before sampling; the legal tokens absorb the freed probability mass.

The reason this took until 2024 to ship in production APIs is performance. Naively, checking every token against the grammar costs ~2.5 million operations per token. The breakthrough (Willard & Louf, 2023, arXiv:2307.09702) was precomputing an index so the legal-token lookup becomes O(1) per step. OpenAI shipped it in August 2024. Anthropic followed with output_config in late 2025, GA in early 2026. If you tried structured output before your provider had constrained decoding and got burned — that’s why. Third-party benchmarks at the time measured 5–10% schema error rates in production without it.

Reality check: 100% schema compliance ≠ 100% correct#

Here’s the first thing that surprised me: teams hear “guaranteed” and stop validating.

The guarantee covers shape, not values. On schema-following benchmarks, current frontier models achieve 97%+ structural compliance — but value-level accuracy runs 15–30 points lower. The JSON always parses. The policy_number field is always a string. Whether it’s the right string is still the model’s problem, and yours.

Constrained decoding moved the failure mode: you no longer debug parsing errors, you debug plausible-looking wrong values. That’s arguably harder, because nothing crashes. Keep your evals. Keep your business-rule validation. The schema guarantee is the floor, not the ceiling.

Wish I knew #1: exclude=True doesn’t do what you think#

Real scenario from our extraction pipeline. Some schema fields are filled by the LLM; others are filled by post-processing — a document UUID from our system, carrier codes from a lookup table. The LLM should never touch those.

The intuitive fix is Pydantic’s exclude=True:

1
class SourceDocument(BaseModel):
2
    filename: str
3
    doc_id: Optional[str] = Field(None, exclude=True)  # ❌ wrong tool

This doesn’t work, and the reason is a distinction worth internalizing: exclude=True affects model_dump() — serialization of an instance. The LLM never sees your instances. It sees model_json_schema() — the description of the class. With exclude=True, doc_id is still in the schema, and the model will happily hallucinate a value for it. In my tests, given a document containing a reference ID, the model invented a fake UUID from it every single time.

The right tool is SkipJsonSchema:

1
from pydantic.json_schema import SkipJsonSchema
2

3
class SourceDocument(BaseModel):
4
    filename: str
5
    doc_id: SkipJsonSchema[Optional[str]] = Field(None)  # ✅ gone from schema

The field disappears from the JSON schema entirely — the LLM can’t fill what it can’t see — but it’s still on the model, so your pipeline code populates it afterward and everything round-trips cleanly. One type annotation, zero hallucinated UUIDs, and a smaller prompt as a bonus.

Wish I knew #2: your schema has a complexity budget#

This one cost me an afternoon of confusion. Our extraction schema grew field by field until one day the API returned: Schema is too complex.

Anthropic enforces a hard limit of roughly 24 optional parameters combined in a structured output schema (grammar compilation cost grows with the number of optional branches). Big flat schemas — exactly what document extraction tends to produce — hit this wall fast.

The fix that worked for us is multi-stage extraction: split the schema by document section, run one extraction call per section with a small focused schema, then merge into the full record — using SkipJsonSchema on every field the pipeline owns. Smaller schemas also mean the model attends to fewer instructions per call, and in practice per-field accuracy went up when we split.

Treat schema size as a design constraint from day one, not a limit you discover in production.

Wish I knew #3: the guarantee can make the model dumber#

The most counterintuitive finding I hit during this research: constrained decoding can hurt output quality.

An EMNLP 2024 paper (“Let Me Speak Freely?”, arXiv:2408.02442) measured it: on reasoning-heavy tasks, format restrictions significantly degrade accuracy. The mechanism makes sense once you see it — the grammar closes off the token paths the model would use to reason in natural language before committing to an answer. You’ve guaranteed the shape of the output by amputating the thinking that produces good output.

The architectural fix is to give the model somewhere to think that the grammar doesn’t reach. Anthropic’s docs state this directly: grammars apply only to the final output, not to thinking blocks — so extended thinking plus structured output lets the model reason freely and then emit schema-compliant JSON:

1
response = client.messages.create(
2
    model="claude-sonnet-4-6",
3
    thinking={"type": "enabled", "budget_tokens": 2000},
4
    output_config={"format": {"type": "json_schema",
5
                              "schema": MySchema.model_json_schema()}},
6
    messages=[...],
7
)

Rule of thumb: extraction and classification → constrain freely. Anything requiring multi-step reasoning → pair the constraint with a thinking budget, or you’re trading correctness for parseability.

The mental model that ties it together#

One more thing that clicked late for me: tool calling and structured output are the same feature. Anthropic’s docs say strict tool use “compiles tool input_schema definitions into grammars using the same pipeline as structured outputs.” One controls what the model says; the other controls how it calls your functions. Same constrained sampling underneath.

Which yields a simple decision guide:

• Real tool invocation, types must be right → strict tool use
• Data extraction / structured replies → structured output (output_config / response format)
• Complex reasoning + structure → extended thinking + structured output
• Schema fields owned by your pipeline, not the LLM → SkipJsonSchema
• Schema growing past ~20 optional fields → split into multi-stage extraction

Structured output is genuinely one of the biggest reliability upgrades in production LLM engineering — we deleted an entire retry-and-repair layer after adopting it. But the guarantee is narrower than the marketing, the sharp edges are real, and every one of these three lessons came from something breaking in production first.

I wish I’d known. Now you do.

I build LLM extraction pipelines for insurance documents. These lessons come from production systems processing real carrier documents at scale.

References#

• Willard & Louf (2023), “Efficient Guided Generation for Large Language Models” — arXiv:2307.09702
• Tam et al. (2024), “Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of LLMs” (EMNLP 2024) — arXiv:2408.02442
• Dong et al. (2024), “XGrammar” — arXiv:2411.15100
• Geng et al. (2025), “JSONSchemaBench” (ICML 2025)
• OpenAI, “Introducing Structured Outputs in the API” (Aug 2024)
• Anthropic docs: Structured outputs / Strict tool use