Your LLM Can Return Good JSON and Nonetheless Be Mistaken

Three weeks after I turned on Structured Outputs for a pipeline that parsed cost affirmation messages into transaction data, I seen that our reconciliation job began flagging a small, regular stream of mismatches.

They weren’t crashes, and never malformed rows both. Simply transactions the place the quantity and sender matched completely however the date was off. One thing like 2 to three% of a given week’s quantity, sufficient to note, not sufficient to be apparent immediately.

At first I assumed that it was a timezone bug. Nevertheless it wasn’t.

After I pulled the uncooked supply messages subsequent to the extracted data, a sample confirmed up: each mismatched transaction got here from a message that by no means talked about a date in any respect.

One thing like “Cost acquired from Chinedu, ₦45,000, ref TXN-82K91.” No date anyplace within the textual content. And the mannequin had crammed transaction_date in anyway, nearly all the time the date the extraction job ran, off by lower than an hour.

The schema mentioned transaction_date: date, required. The mannequin could not return nothing. So it did not.

I would been treating “the JSON is legitimate” because the end line for this pipeline, and for weeks it appeared like one.

It is not.

It is the purpose the place a quieter form of failure turns into potential, one which by no means throws an error or fails a kind test, and does not present up till one thing downstream will depend on the worth being actual.

Most of what will get written about Structured Outputs stops at “it may’t return damaged JSON anymore,” as if that settles the reliability query. It settles one model of it.

The lure of the right schema

Structured Outputs clear up an actual downside. Earlier than native schema enforcement, getting dependable JSON out of an LLM meant regex parsers, retry loops, and prompts that principally begged the mannequin: “ONLY output JSON, no markdown, no preamble.”

With the fashionable OpenAI Python SDK and a Pydantic mannequin, most of that class of ache simply goes away:

import loggingfrom datetime import datefrom pydantic import BaseModelfrom openai import OpenAIlogger = logging.getLogger(__name__)shopper = OpenAI()class Transaction(BaseModel):    sender: str    quantity: float    transaction_id: str    transaction_date: datedoc = """Cost acquired from Chinedu.Quantity: ₦45,000Reference: TXN-82K91Date: 11 August 2026"""# gpt-4o-mini stored merging the quantity and reference into one area on# messages with uncommon formatting, so this stays on the complete mannequin# regardless of the fee. Price revisiting as soon as mini catches up.completion = shopper.beta.chat.completions.parse(    mannequin="gpt-4o",    messages=[        {"role": "system", "content": "Extract the transaction details."},        {"role": "user", "content": document},    ],    response_format=Transaction,)txn = completion.decisions[0].message.parsedlogger.information("parsed txn %s", txn.transaction_id)

Run it towards a clear message and it really works precisely as marketed. Each key current, each kind appropriate, no attempt/besides wanted simply to catch a stray markdown fence across the JSON.

Then somebody forwards you a message like this one:

doc = """Cost acquired from Chinedu.Quantity: ₦45,000Reference: TXN-82K91"""# No date on this one.

however the schema does not care that the date is not there. It is nonetheless marked required, so one thing has to fill that slot, and it is by no means going to be the schema that bends.

The mannequin reaches for no matter will get it to a sound worth as an alternative: the present date, the coaching cutoff, a plausible-looking guess.

What comes again type-checks completely. It is also fully made up, and there is nothing within the response itself that tells you which ones fields are which.

Designing schemas for uncertainty

The repair is a psychological shift greater than a code change. An empty area is not an error in extraction, it is usually simply the reality. Making fields nullable takes the stress off the mannequin to invent one thing:

class Transaction(BaseModel):    sender: str | None    quantity: float | None    transaction_id: str | None    transaction_date: date | None

Now if the date’s lacking, the mannequin can simply say so. This additionally brings a distinction that is simple to blur, which is extraction versus inference.

Extraction is “inform me precisely what’s within the textual content.” Whereas inference is “inform me what it implies.” A message that claims “paid on Tuesday” and a schema demanding an ISO date, that is inference, whether or not you meant to ask for it or not.

Typically inference is strictly what you need, however the resolution ought to be yours, not one thing the mannequin makes for you by default. A nullable area arms that call again to your personal code:

if transaction.transaction_date is None:    request_missing_info(transaction_id=transaction.transaction_id)

Proof and provenance

Nullable fields repair the “inventing values from nothing” downside. They do not repair the opposite one, which is truthfully worse: the mannequin offers you a worth, and you haven’t any option to inform if it really learn that worth off the web page or pattern-matched its manner there.

With a standard chat response you’ll be able to at the very least watch it cause its option to a solution. Structured Outputs skip straight to the ultimate type. So I began asking for a second area alongside each worth, the precise chunk of supply textual content that supposedly backs it up.

from pydantic import Disciplineclass Extracted(BaseModel):    """Generic wrapper so I am not writing a near-identical class per area kind."""    worth: float | date | str | None    proof: str | None = Discipline(description="precise quote backing this worth, empty if not discovered")class Transaction(BaseModel):    sender: str | None    quantity: Extracted    transaction_id: str | None    transaction_date: Extracted

The generic Extracted wrapper is a shortcut, not a finest apply. worth is now a union kind as an alternative of a clear float, which prices among the kind security the unique schema had.

That commerce is price it as soon as a schema has greater than a few area varieties, writing ExtractedFloat, ExtractedDate, ExtractedString individually is simply busywork at that time. For one or two fields, hold the precise lessons, they’re normally cleaner.

The sample earns its hold two methods. Ordering proof earlier than worth issues as a result of keys generate in sequence, so the mannequin has to jot down down what it is taking a look at earlier than committing to a solution, a small pressured show-your-work.

And it offers a reviewer one thing concrete to test with out re-reading the supply. If worth is crammed in however proof is empty, or accommodates textual content that is not within the supply anyplace, that mismatch is the hallucination displaying up within the information itself.

It does not come free. On a batch of some hundred transaction messages, including proof fields throughout the schema pushed output tokens up by roughly a 3rd, and latency rose sufficient to matter at pipeline scale.

Not price it for a five-digit zip code. However for a monetary determine somebody’s going to behave on, it is positively price it.

The boundary between technology and validation

By this level the schema is carrying so much: nullable varieties so it does not invent issues, proof fields so I can catch it when it does anyway.

However there’s a complete class of wrongness neither of these touches, which is whether or not the worth makes any sense as a truth in regards to the world.

The schema ensures quantity is a float. It says nothing about whether or not that float is adverse, or whether or not transaction_date is someway three days from now.

Early on I attempted fixing this within the immediate, with directions like “the quantity should be higher than zero,” which in hindsight was a wierd factor to ask a language mannequin to implement. It is not a calculator. A validator does this precisely proper, each single time, without cost:

from pydantic import model_validator, ValidationErrorclass Transaction(BaseModel):    sender: str | None    quantity: float | None    transaction_id: str | None    transaction_date: date | None    @model_validator(mode="after")    def check_sane_values(self) -> "Transaction":        # adverse quantities have proven up precisely twice, each occasions as a result of        # the supply message described a refund, not a cost        if self.quantity isn't None and self.quantity <= 0:            elevate ValueError(f"quantity should be optimistic, obtained {self.quantity}")        if self.transaction_date isn't None and self.transaction_date > date.right now():            elevate ValueError(f"transaction_date {self.transaction_date} is sooner or later")        return self

So now the API is guaranteeing construction the second it generates the response, and Pydantic is guaranteeing the information is sensible the second it is parsed into the article, the identical manner each time, no LLM concerned in that second test in any respect.

When the validator throws, you’ve got obtained choices: kick the report to a human, or hand the precise error again to the mannequin and let it attempt once more. I went with the second, capped exhausting at two retries:

MAX_RETRIES = 2def extract_with_retry(doc: str) -> Transaction:    historical past = [        {"role": "system", "content": "Extract the transaction details."},        {"role": "user", "content": document},    ]    for try in vary(MAX_RETRIES + 1):        completion = shopper.beta.chat.completions.parse(            mannequin="gpt-4o", messages=historical past, response_format=Transaction        )        uncooked = completion.decisions[0].message.content material        attempt:            return Transaction.model_validate_json(uncooked)        besides ValidationError as e:            if try == MAX_RETRIES:                elevate  # hand over, let the caller route this to a human            logger.warning("validation failed on try %d: %s", try, e)            historical past += [                {"role": "assistant", "content": raw},                {"role": "user", "content": f"That failed validation: {e}. Fix only the bad field."},            ]

The MAX_RETRIES cap really issues greater than it seems to be. My first intuition was to let it hold making an attempt, which is a mistake. Two failed makes an attempt nearly all the time means the supply doc is the precise downside, not the immediate, and a 3rd automated go simply burns API calls on one thing a human clears in ten seconds.

None of that is OpenAI-specific both, despite the fact that each code block right here is. Swap in Anthropic’s software use or a self-hosted setup with vLLM and Outlines and the Pydantic mannequin does not transfer an inch, it is simply the API name round it that modifications.

After I first obtained this working, my bar for achievement was embarrassingly low: did the mannequin fill out the article with out breaking my parser.

Wanting again, that bar rewards the incorrect factor solely, as a result of a mannequin that eagerly fills each area no matter what’s really in entrance of it is not dependable. It is simply assured, which is a unique and extra harmful factor.

Structured Outputs are genuinely good at what they do. They only do not do the factor I initially thought they did. They assure form, not fact, and when you cease worrying about brackets and quote escaping, the actual query continues to be sitting there ready: does each worth on this object have an precise cause to exist?

That query was all the time the exhausting half. The schema simply used to cover it from me.