Administrator
Published on 2026-11-16 / 5 Visits
0
0

LLM Structured Output Validation: Catch Schema-Valid but Wrong JSON

To validate structured LLM output, check the response status, parse the JSON, enforce the object contract, then verify values against business rules and an independent source. A valid object is a candidate piece of data. It is not permission to execute an action.

The useful test is not “can the model produce JSON?” It is “will our application reject a plausible, correctly typed, internally consistent lie?”

Here is a small, runnable answer. It uses only Python's standard library and no model API. Its twelve fixtures test the application boundary, not model accuracy.

Start with a failure that a schema cannot settle

Suppose a trusted source record contains an invoice identifier, currency and three amounts in integer cents. The model returns the same identifier and currency, but doubles the net amount and adjusts the total so the arithmetic still balances.

Every field has the expected type. The total equals net plus tax. Yet the record disagrees with the source.

A schema describes constraints on an instance; required fields and additional-property rules help enforce the object contract. See the JSON Schema object reference. Establishing that an extracted amount belongs to the intended document requires more than that structural contract.

MLflow's structured-output guide similarly distinguishes format enforcement from correctness of field values. The example below concentrates on the boundary after generation, so it does not depend on a particular provider's current schema subset.

A runnable validator and twelve adversarial fixtures

Save this as validation_example.py and run python3 validation_example.py.

The fixture assumes a simple document with net, tax and total in one currency, with no discounts, rounding adjustment or credit-note semantics. The source record is an application-supplied, independently verified reference. It must never be copied from the output being checked.

"""Deterministic teaching fixture, not an invoice or accounting system."""
import json

def reject_constant(value):
    raise ValueError("non-finite number")

def unique_object(pairs):
    out = {}
    for key, value in pairs:
        if key in out:
            raise ValueError("duplicate key")
        out[key] = value
    return out

def validate(raw, source, response_complete=True):
    # source is a trusted fixture supplied by the application, never by the model.
    if not response_complete:
        return "review:response"
    if len(raw.encode("utf-8")) > 4096:
        return "reject:size"
    try:
        obj = json.loads(raw, object_pairs_hook=unique_object,
                         parse_constant=reject_constant)
    except (ValueError, RecursionError):
        return "reject:parse"
    expected = {"document_id", "currency", "net_cents", "tax_cents", "total_cents"}
    if type(obj) is not dict or set(obj) != expected:
        return "reject:shape"
    if any(type(obj[k]) is not str for k in ("document_id", "currency")):
        return "reject:type"
    for k in ("net_cents", "tax_cents", "total_cents"):
        if type(obj[k]) is not int or not 0 <= obj[k] <= 100000000:
            return "reject:type"
    if obj["document_id"] != source["document_id"]:
        return "review:document"
    if obj["currency"] != source["currency"]:
        return "review:currency"
    if obj["net_cents"] + obj["tax_cents"] != obj["total_cents"]:
        return "review:arithmetic"
    if any(obj[k] != source[k] for k in expected):
        return "review:source"
    return "accept:validated-data"

if __name__ == "__main__":
    source = dict(document_id="INV-17", currency="USD",
                  net_cents=1000, tax_cents=200, total_cents=1200)
    cases = [
        ("correct", json.dumps(source), True, "accept:validated-data"),
        ("wrong document", json.dumps(dict(source, document_id="INV-18")), True, "review:document"),
        ("wrong currency", json.dumps(dict(source, currency="EUR")), True, "review:currency"),
        ("wrong arithmetic", json.dumps(dict(source, total_cents=1300)), True, "review:arithmetic"),
        ("balanced but invented", json.dumps(dict(source, net_cents=2000, total_cents=2200)), True, "review:source"),
        ("boolean integer", json.dumps(dict(source, tax_cents=True)), True, "reject:type"),
        ("duplicate key", '{"document_id":"a","document_id":"b"}', True, "reject:parse"),
        ("non-finite", '{"total_cents":NaN}', True, "reject:parse"),
        ("truncated", '{"document_id":', False, "review:response"),
        ("extra key", json.dumps(dict(source, approved=True)), True, "reject:shape"),
        ("missing key", json.dumps({k:v for k,v in source.items() if k != "currency"}), True, "reject:shape"),
        ("oversize", " " * 4097, True, "reject:size"),
    ]
    for name, raw, complete, expected in cases:
        actual = validate(raw, source, complete)
        assert actual == expected, (name, actual, expected)
    print(f"{len(cases)} deterministic fixtures passed; no model API called.")

The example was executed locally on November 16, 2026. Its output was:

12 deterministic fixtures passed; no model API called.

This is evidence that these branches behave as described on the fixed fixtures. It is not a benchmark of any LLM, evidence of production accuracy, or a complete implementation of JSON Schema. The explicit type/key checks implement only this example's small object contract.

The parsing choices are deliberate. Python's JSON decoder can accept repeated object names and non-finite numeric constants by default. Its documentation also warns about resource use for untrusted input. The example rejects duplicate keys and non-finite constants and bounds input size; a production service additionally needs request, concurrency and parsing-resource limits. See the Python JSON documentation.

Why the balanced-but-invented fixture matters

The first semantic check binds the answer to a document. The second binds the currency. Arithmetic is checked next. Only then do we compare the values with the reference.

That ordering gives useful failure categories, not just a boolean result. The balanced-but-invented fixture passes arithmetic and fails review:source. A test suite containing only malformed JSON and bad totals would miss this failure.

The strict integer check also excludes Python booleans. This is a local implementation detail worth testing rather than relying on a permissive conversion that turns true into a number.

Using integer cents avoids binary-floating-point arithmetic in this narrow example. It does not solve currency precision generally. Configure the unit and rounding rules for the actual currency and document type; don't reuse the two-decimal assumption for every financial record.

Where does an independent source come from?

The model should not manufacture both an answer and the evidence used to approve it.

For an extraction workflow, keep the original document identity and immutable revision, along with page or span locators. A deterministic lookup or separately verified reference can supply the expected fields. For new documents where no verified reference exists, route disputed or consequential values to review instead of treating a second model's agreement as ground truth.

Exact matching is intentionally narrow. OCR errors, alternate formatting and legitimate normalization can create false rejections. Resolve those through documented normalization rules and fixtures. The correct response is not to relax checks until every example passes.

Derived fields require their own rules. A category or summary cannot be validated by the invoice equality check. Use task-specific evidence criteria, representative held-out cases and, where necessary, independent human judgment.

Route failures according to cause

Outcome Appropriate next step
Incomplete response Inspect the adapter's completion/refusal state; repair the cause before retrying
Parse, shape or type failure Fix the contract or request; allow bounded regeneration only when meaningful
Wrong document or currency Inspect routing and source binding; keep downstream writes blocked
Arithmetic disagreement Inspect extraction and the document's accounting rules
Source disagreement Obtain evidence or review; another plausible answer is insufficient
Validated data Pass data to the next stage, which still checks authorization and execution safety

response_complete is a simplified adapter input. Real integrations must separately handle refusal, truncation and transport errors using their actual SDK response fields. This article deliberately does not invent a universal field name.

The example performs no retries, writes or payments. If validated data later drives an agent action, the execution layer still needs authorization, idempotency and result reconciliation. The site's payment intent and fulfillment receipts discussion covers that separate boundary.

What to carry into production

Keep the schema version, source revision, validation category and final disposition together. Store sensitive source material under appropriate access and retention controls rather than copying complete documents into unrestricted logs.

Add fixtures from actual failures: wrong document revisions, missing evidence, legitimate rounding, unsupported currencies and adversarial text. Keep the deterministic validator suite separate from live extraction evaluation. One checks your guardrails; the other measures how often generated candidates satisfy the intended task.

The practical improvement is a testable acceptance boundary: a response can be well-formed, arithmetically consistent and still be rejected for the right reason.


Comment