Stop asking the model for JSON

There was a period, not that long ago, when getting structured data out of a language model meant writing a prompt that ended with "Respond in JSON" and then praying. The prayers failed a lot. The model would wrap the JSON in markdown fences, or add a preamble, or insert a trailing comma, or decide that one of the fields should be a nested object with a different name. Every one of those failures was a small production incident, and they compounded into a general distrust of the whole idea.

Then the providers shipped structured output modes and function calling, and a lot of teams breathed a sigh of relief and stopped thinking about the problem. That is the wrong response. The constraint mechanism removed the obvious failure modes, but it did not remove the need for a discipline around them. The model will still lie. It will just lie in valid JSON now.

What the constraint actually guarantees

When a model API offers structured output, it guarantees that the tokens it produces will form a valid instance of your schema. No fences, no preambles, no trailing commas. That is a real improvement, and it removes an entire category of parsing bugs from your codebase. The model cannot produce invalid JSON, because the sampling process is constrained to only emit tokens that keep the output valid.

Here is the thing it does not guarantee: that the values inside the JSON are correct, sensible, or even present in the intended spirit. The model can produce valid JSON where a date is the string "tomorrow", where a required email is empty, where a numeric field contains a number that contradicts the text it was summarizing. Validity and correctness are different properties, and the constraint only bought you the first one.

Validation is the non-negotiable layer

I have never seen a production system that was improved by trusting the model's output directly into a database. The output needs to pass through a validation layer that treats the model like an unreliable external service, because that is what it is.

Use a schema library and declare the expected shape before you even make the call. This does two jobs. First, it defines the contract for the API call itself, so the structured output mode knows what to produce. Second, it becomes the gatekeeper after the call, so anything that does not match fails loudly instead of flowing into the system as a subtly wrong record.

The failures you are actually guarding against are not the parse failures anymore. They are the semantic ones. The model returns a status field with value "active" when the source document says the account is suspended. The validation schema says status is a string, so it passes. This is where the real engineering lives: custom checks beyond the type level. Enum validation, range checks, cross field consistency, a check that the returned identifier actually exists in your database. The schema catches shape errors. Your business rules catch meaning errors. You need both.

Repair, do not just fail

When validation fails, the least useful thing you can do is surface a generic error to the user. The model produced something close to what you wanted, and close is recoverable.

The pattern that works is a correction loop. Pass the schema and the validation error message back to the model and ask it to fix the output. Most of the time it will. Models are quite good at reading a specific error like "field amount must be a positive number, got -12" and producing a corrected version, because the error message is far more information than the original prompt had.

You want to bound this loop. One retry handles the vast majority of cases. Two is a reasonable ceiling. After that, the model is stuck and you should treat the call as failed rather than spinning. I have also seen teams skip the repair loop entirely by generating several candidate outputs at once and picking the one that validates, which is a fine trick when the task is short and the failure rate is low. The point is that a single unvalidated pass is not a system. It is a coin flip.

Design the schema for the model, not for the database

This is the part that surprises people. The schema you give the model and the schema you store in your database are not necessarily the same shape, and trying to make them identical usually makes the model's job harder.

A few rules that have held up for me. Use names that describe what the value is, not what the column is called. A field called "customer_reference" is clearer to a model than "cr". The model reads your schema as documentation. Make the documentation good. Keep the schema flat where possible. Deeply nested structures are easier for the model to get wrong, because the model has to keep more context in mind while generating. Add descriptions to the fields where the semantics are not obvious. If the value should be a date in ISO format, say so in the description, because the model will otherwise guess a format.

Most important: make the model's job boring. The harder the extraction task, the more likely the output is wrong. If you are asking the model to extract a nested structure with inference about meaning, split it into smaller extractions. A chain of small, simple structured outputs beats one giant ambitious one, and the error messages from validation are easier to turn into successful retries.

Defaults and missing data are decisions

An extraction will sometimes return no value for a field that exists. The source text did not contain it. The model's job is to say so honestly, not to invent something plausible.

Make "unknown" a first class value in your schema. Give the model an explicit way to represent absence, whether that is null, an empty string, or a dedicated sentinel. The alternative is the model filling in a plausible guess, and a plausible guess stored in your database is a time bomb. When the user asks why the record says the delivery was on the 14th and the source says nothing about a date, the answer is that a model guessed. Decide up front whether guesses are acceptable in your domain. For most domains they are not, and the schema should make it easier to be honest than to be wrong.

The practical checklist

The system I trust looks like this. Declare the schema first, in code, as the single source of truth. Request structured output for that schema. Validate everything, types first, then business rules. On failure, repair with the error message, bounded to one or two retries. Represent missing data explicitly. Log every failed validation, because a high repair rate is a signal that your schema or your task design is wrong, and you want to notice that before your users do.

Treat the model as a junior contractor: capable of doing the work, unreliable in the details, and in desperate need of a review layer. Review layers are how software stays correct. This is no different.