Your pipeline expects structured JSON. What arrives is triple backticks, a cheerful preamble, three nulled fields, and a “helpful” extra field. Syntax is valid. Your parser shrugs. Your downstream code explodes. Friday’s deploy becomes Saturday’s incident.
JSON mode was supposed to fix this. OpenAI constrains at the token level. Anthropic still fails 3-4% of the time. Gemini loops infinitely below temperature 1.0. This guide maps the differences with working cross-provider code — one wrapper, every quirk isolated.
OpenAI: response_format with Strict Mode
OpenAI’s implementation is the gold standard. response_format: {"type": "json_schema", "json_schema": {...}} with strict: true (see OpenAI’s structured output docs) compiles your schema into a finite state machine. The model is constrained at the token level — it cannot generate a token that would produce invalid JSON. The result: 98%+ valid JSON rate in production.
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-5.5",
messages=[{"role": "user", "content": "Extract: John Doe, john@example.com, $49.99"}],
response_format={
"type": "json_schema",
"json_schema": {
"name": "customer_extract",
"strict": True,
"schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string", "format": "email"},
"amount": {"type": "number"}
},
"required": ["name", "email", "amount"],
"additionalProperties": False
}
}
}
)The additionalProperties: false flag is the unsung hero. Without it, the model can add extra fields. With it, those fields are blocked at the token level. Always include it when your downstream parser expects exact schema matching.
When sending structured data through any API, follow security best practices for handling keys, PII, and access controls.
Anthropic: output_config.format
Anthropic’s structured output uses output_config.format — functional, but with two important constraints. First, you cannot combine structured output with citations. If your use case requires both JSON output and source attribution, you need to embed citations as a field within the JSON schema rather than using Anthropic’s native citation feature. Second, with extended thinking enabled, JSON generation can occasionally produce minor deviations from strict schema adherence — the thinking process introduces variability that constrained decoding doesn’t fully eliminate.
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-20250514",
messages=[{"role": "user", "content": "Extract: John Doe, john@example.com, $49.99"}],
output_config={
"format": "json_schema",
"json_schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"},
"amount": {"type": "number"}
},
"required": ["name", "email", "amount"]
}
}
)Valid JSON rate in production: ~96-97%. The gap versus OpenAI’s 98%+ is small but real. Mitigate by adding a lightweight post-processing layer that validates and retries on parse failure.
Google Gemini: JSON Schema via GenerationConfig
Gemini’s structured output integrates cleanly with its broader GenerationConfig. A key advantage: you can combine JSON Schema with tool calling — a capability neither OpenAI nor Anthropic fully supports. The schema constrains the output format while tools handle function execution in parallel.
import google.generativeai as genai
model = genai.GenerativeModel(
"gemini-3.1-pro",
generation_config={
"response_mime_type": "application/json",
"response_schema": {
"type": "object",
"properties": {
"name": {"type": "string"},
"email": {"type": "string"},
"amount": {"type": "number"}
},
"required": ["name", "email", "amount"]
}
}
)⚠️ Critical: Keep temperature at 1.0 on Gemini. Setting it below 1.0 with JSON Schema enabled can trigger looping behavior — the model repeatedly tries to satisfy the schema constraint but reduced temperature limits its ability to explore token alternatives. The fix is counterintuitive if you’re coming from OpenAI: higher temperature = more reliable structured output on Gemini.
Cross-Provider Wrapper
Define your schema once. Generate across providers. The wrapper handles provider-specific format differences so your application code doesn’t have to.
from typing import TypeVar, Generic
from pydantic import BaseModel
T = TypeVar('T', bound=BaseModel)
class UnifiedStructuredOutput(Generic[T]):
"""Generate structured output from any provider using one schema."""
def __init__(self, model: str, client, provider: str):
self.model = model
self.client = client
self.provider = provider # "openai", "anthropic", or "gemini"
async def generate(self, prompt: str, schema: type[T]) -> T:
if self.provider == "openai":
return await self._openai_generate(prompt, schema)
elif self.provider == "anthropic":
return await self._anthropic_generate(prompt, schema)
elif self.provider == "gemini":
return await self._gemini_generate(prompt, schema)
async def _openai_generate(self, prompt: str, schema: type[T]) -> T:
response = await self.client.chat.completions.create(
model=self.model,
messages=[{"role": "user", "content": prompt}],
response_format={
"type": "json_schema",
"json_schema": {
"name": schema.__name__,
"strict": True,
"schema": schema.model_json_schema()
}
}
)
return schema.model_validate_json(response.choices[0].message.content)One schema definition. Three providers. The wrapper isolates each provider’s API quirks behind a uniform interface — when Anthropic changes its structured output API, you update one file, not every call site. For validating that your cross-provider wrapper survives model version upgrades, our CI/CD testing guide covers automated schema validation and contract testing.
For routing structured output requests across multiple models in a production application, our multi-model application architecture covers the provider-agnostic routing layer.
When Structured Output Fails Silently
Missing fields. The model omits a required field because the input didn’t contain enough information to populate it. Fix: make fields optional in the schema when the input data may not contain them. Required should mean “the model will always generate this” — not “the model will guess if data is missing.”
Null handling. A field is marked required. The input data is ambiguous. The model generates null — technically present, functionally useless. Fix: add explicit instructions in the prompt: “If a field cannot be determined, omit it entirely rather than setting it to null — and mark the extraction as incomplete.”
Extra fields. Without additionalProperties: false (OpenAI) or equivalent schema strictness, the model can add fields you didn’t ask for. These silently pass JSON validation but crash downstream parsers that expect exact schema matching.
When Structured Output Breaks in Production
The docs make structured output sound airtight. Define a schema. Get valid JSON. Ship. Here is what the docs do not tell you.
The silently accepted null that passed validation. A fintech team built an invoice extraction pipeline. Schema required invoice_number: {type: "string"}. The model received a scanned document where the invoice number was blurry. It returned "invoice_number": null. Required field. Present. Valid JSON. Null. The downstream accounting system accepted the record without complaint — null wasn’t invalid, it was just useless. Three weeks and 1,400 invoices later, the finance team reported missing invoice numbers in their quarterly reconciliation. The structured output was “valid.” The business outcome was wrong. Fix: add a post-extraction validation layer that checks not just schema compliance but semantic completeness — if a field was declared required and the model returned null, flag it for human review.
The temperature=0 Gemini infinite loop. A developer migrated their document classification pipeline from GPT-5.5 to Gemini 3.1 Pro. Everything worked in testing — five test documents, five correct JSON outputs. They deployed to production on Friday at 4 PM. By 4:17 PM, their Kubernetes pods were OOMKilled. Gemini was generating 50,000+ output tokens per request — all of them discarded, regenerated, discarded again. The cause: temperature=0 with JSON Schema. At zero temperature, Gemini’s token selection becomes deterministic. When the deterministic path hits a schema constraint it cannot satisfy (a required enum field with no valid match in the document), it loops — generates, validates against schema, fails, retries the same tokens, fails again. The fix was setting temperature to 1.0. The cost was $1,400 in wasted API credits during a 17-minute incident. The lesson: “deterministic” does not mean “safe” on every provider.
The Anthropic citation + JSON incompatibility. A legal tech startup used Claude to summarize deposition transcripts and return structured JSON with cited source lines. Their prompt was clean: transcript in context, then instructions to extract key facts as JSON with source_lines: [int]. It worked for six weeks. Then they upgraded from Claude Sonnet 3.5 to Claude Sonnet 4 and attempted to turn on native citations via the citations output config. The API rejected every request. Anthropic’s output_config.format: "json_schema" does not combine with citations.enabled: true. The feature they assumed would improve accuracy silently blocked their entire pipeline. They reverted the change in 20 minutes, but only after their on-call engineer spent 90 minutes debugging why “the same code that worked yesterday” suddenly returned 400 errors. Fix: check your provider’s structured output compatibility matrix before combining features. OpenAI’s structured output blocks function calling. Anthropic’s JSON mode blocks citations. Gemini lets you combine both but breaks below temperature 1.0. The capabilities table you need isn’t in any one provider’s docs — you have to build it from incident reports like this one.
The extra field that broke a database migration. A content platform used GPT-4 with response_format to extract article metadata: title, author, publish_date, tags. Six months of perfect extraction. Then they upgraded to GPT-5.5 and their database migration failed. GPT-5.5 had started adding a "language": "en" field to every extraction — a field that didn’t exist in their 2024 schema definition. The field was helpful. It was accurate. It passed JSON validation. And it crashed their ETL pipeline because the database column didn’t exist. The original schema had no additionalProperties: false. The model was being helpful. The pipeline was unforgiving. Fix: always use additionalProperties: false on OpenAI. Always validate the output JSON against your exact schema in the application layer, not just the API layer. A model upgrade changes behavior in ways that pass API validation but break your database.
FAQ
Which provider has the most reliable JSON mode?
OpenAI with strict mode. 98%+ valid JSON rate. Constrained decoding at the token level means the model is physically incapable of generating invalid JSON. Anthropic is close at 96-97% but lacks true constrained decoding. Gemini is reliable at temperature 1.0 but degrades at lower temperatures.
Does constrained decoding add latency?
OpenAI strict mode adds negligible overhead — the finite state machine runs in parallel with token generation. Anthropic’s JSON mode adds no measurable latency. Gemini’s overhead is minimal at temperature 1.0. Overall: structured output does not meaningfully impact generation speed on any provider.
Can I reuse the same JSON Schema across all three providers?
The schema definition — types, properties, required fields — is portable. The API syntax for providing the schema differs per provider. The UnifiedStructuredOutput wrapper above handles the translation. One schema definition, three API formats, one uniform output.
Structured output vs. function calling — which should I use?
Structured output: use when you need the model to return data in a specific format. Function calling: use when you need the model to trigger an external action. They’re complementary — structured output defines the shape of the response. Function calling defines the action to take. On Gemini, you can use both simultaneously. On OpenAI and Anthropic, they’re mutually exclusive per request. For the complete implementation guide covering tool definitions, schema design, and provider differences, see our function calling and tool use comparison.
Should I use structured output for every LLM call, or only when I need JSON?
Use it whenever you need the response parsed by code — which in production means almost every call. Human-readable text is a special case (chat interfaces, email drafts, content generation). Everything else — classification labels, entity extraction, search filters, routing decisions, API parameter construction — benefits from structured output. The latency overhead is negligible. The reliability gain is massive. The one caveat: if your use case genuinely needs free-form creative text (marketing copy, story generation), structured output constrains the model in ways that degrade creativity. For those cases, generate raw text and apply post-hoc parsing. For everything else, generate structured output and skip the parsing step entirely. For troubleshooting common issues like schema validation errors and unexpected null fields, consult the FAQ section above.
The quality of your structured output isn’t a function of how well you write prompts. It’s a function of which provider you’re calling, which schema constraints you’ve enabled, and whether you’ve wrapped provider-specific quirks behind a uniform interface.
Start with OpenAI strict mode if you need maximum reliability. Use the cross-provider wrapper if you route between models. And if you’re on Gemini, keep that temperature at 1.0 — your JSON parser will thank you.
Consistent JSON from GPT, Claude, and Gemini — without three different parsers. Test structured output on TokSpan — send the same JSON schema to every provider through one endpoint and see which one breaks first. $5 free to start.