Structured Outputs From LLMs: A Practical Engineering Guide
Learn how structured outputs and schema enforcement make AI engineering reliable — get LLM responses that feed safely into downstream code without parsing failures.
Parsing an LLM response in production code is a gamble you'll eventually lose. The model returns JSON that's slightly malformed, wraps the payload in markdown fences, or adds a friendly sentence before the object you needed. Your JSON.parse call throws. Your pipeline stops. Your user sees an error. This isn't a model quality problem. It's an architecture problem.
Structured outputs are the engineering answer. They constrain the model's generation at the decoding layer so the output conforms to a schema you define, rather than hoping a prompt instruction holds under load. The difference between a prompt that says "respond in JSON" and a schema-enforced output is the same as the difference between asking a contractor to measure twice and physically handing them a template they can't deviate from.
We've hit this wall on enough production systems at Laxaar to have strong opinions about it. Prompt-only formatting works in demos. It fails in production at a rate that correlates directly with request volume. At a few hundred daily calls, you notice the odd failure. At tens of thousands, it becomes a reliability incident.
What you'll learn
- Why prompt-only formatting fails under load
- Constrained decoding vs schema validation
- Using JSON Schema with OpenAI and Anthropic APIs
- Pydantic and structured output in Python
- Zod and structured output in TypeScript
- Designing schemas that models can actually follow
- Retry, fallback, and partial-parse strategies
- Frequently Asked Questions
Why prompt-only formatting fails under load
Prompt-only formatting is an instruction to the model, not a contract. Models comply most of the time. "Most of the time" is not a software reliability target.
The failure modes are predictable. Models add explanatory prose before the JSON object because their training data is full of that pattern. They wrap output in markdown code fences (```json) because that's how JSON appears in most documentation. They generate a trailing comma in the last array element, which is valid JavaScript but invalid JSON. They omit optional fields when the context doesn't strongly suggest them, even when your schema marks them required.
Each of these is a parsing failure. Each one means downstream code either crashes or has to implement defensive parsing that grows into its own maintenance burden.
The subtler problem is that prompt-only formatting degrades under the very conditions where correctness matters most: high context load, complex nested outputs, and edge-case inputs. A prompt that reliably produces clean JSON on a simple two-field extraction starts failing on a twenty-field document analysis. The more your schema needs, the less reliable instruction-following becomes without enforcement.
This is the honest trade-off: schema enforcement adds latency (a few milliseconds for validation, sometimes more for constrained decoding), and it constrains the model's output vocabulary. For creative or conversational tasks, that constraint is wrong. For data extraction, classification, and any output that feeds code, it's the only sensible default.
Constrained decoding vs schema validation
These are two distinct approaches and it's worth being precise about the difference.
Constrained decoding is grammar-level enforcement at inference time. The model's token sampling is filtered so only tokens that advance a valid parse of the target schema are allowed. The output is guaranteed to be valid JSON (or whatever format you've specified) because invalid tokens are never selectable. This is what OpenAI's response_format: { type: "json_schema" } mode and Anthropic's structured output feature implement. The model literally cannot generate malformed output.
Schema validation is post-generation enforcement. The model generates freely, then the output is parsed and validated against a schema. If validation fails, you retry, fallback, or raise an error. This is what most library-level approaches (Pydantic, Zod) do when they parse a string response into a typed object.
Constrained decoding is stronger. It eliminates an entire class of parse failures before they happen. Schema validation is more flexible (it works with any model, including self-hosted ones that don't support native constrained decoding), but it requires a retry strategy.
In practice, the Laxaar team uses constrained decoding when the API supports it and schema validation as a second line of defense, not a replacement.
Using JSON Schema with OpenAI and Anthropic APIs
Both OpenAI and Anthropic expose first-class structured output support. The implementation differs slightly between them.
OpenAI uses response_format with a full JSON Schema definition:
from openai import OpenAI
import json
client = OpenAI()
schema = {
"type": "object",
"properties": {
"sentiment": {
"type": "string",
"enum": ["positive", "negative", "neutral"]
},
"confidence": {
"type": "number",
"minimum": 0,
"maximum": 1
},
"key_phrases": {
"type": "array",
"items": {"type": "string"},
"maxItems": 5
}
},
"required": ["sentiment", "confidence", "key_phrases"],
"additionalProperties": false
}
response = client.chat.completions.create(
model="gpt-4o-2024-11-20",
messages=[
{"role": "user", "content": "Analyze the sentiment of: 'The deployment was rocky but the team recovered well.'"}
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "sentiment_analysis",
"strict": True,
"schema": schema
}
}
)
result = json.loads(response.choices[0].message.content)
Anthropic uses a tool-calling pattern where a single tool definition acts as the output schema. The model is forced to call that tool, producing a structured argument object:
import anthropic
import json
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
tools=[{
"name": "record_sentiment",
"description": "Record the sentiment analysis result",
"input_schema": {
"type": "object",
"properties": {
"sentiment": {
"type": "string",
"enum": ["positive", "negative", "neutral"]
},
"confidence": {"type": "number"},
"key_phrases": {
"type": "array",
"items": {"type": "string"}
}
},
"required": ["sentiment", "confidence", "key_phrases"]
}
}],
tool_choice={"type": "tool", "name": "record_sentiment"},
messages=[{"role": "user", "content": "Analyze the sentiment of: 'The deployment was rocky but the team recovered well.'"}]
)
result = response.content[0].input
The tool_choice parameter with a specific tool name is what forces the model to produce a structured tool call rather than a free-form response.
Pydantic and structured output in Python
Pydantic's integration with LLM libraries is the most ergonomic structured output pattern in Python. Define a model, pass it to the client, get a typed object back.
from pydantic import BaseModel, Field
from typing import Literal
from openai import OpenAI
client = OpenAI()
class ExtractionResult(BaseModel):
company_name: str = Field(description="The legal name of the company")
founding_year: int = Field(ge=1800, le=2026)
headquarters_city: str
industry: Literal["SaaS", "Fintech", "Healthcare", "E-commerce", "Other"]
employee_count_range: Literal["1-10", "11-50", "51-200", "201-1000", "1000+"]
completion = client.beta.chat.completions.parse(
model="gpt-4o-2024-11-20",
messages=[
{"role": "user", "content": "Extract company info from: 'Stripe, founded in 2010, is a fintech company headquartered in San Francisco with over 8,000 employees.'"}
],
response_format=ExtractionResult,
)
result: ExtractionResult = completion.choices[0].message.parsed
print(result.company_name) # "Stripe"
print(result.founding_year) # 2010
The parse method on client.beta.chat.completions handles schema generation and response parsing automatically. If parsing fails, it raises a ValidationError with field-level detail: far more debuggable than a generic JSONDecodeError.
For Anthropic, the instructor library provides the same Pydantic-native experience:
import instructor
import anthropic
from pydantic import BaseModel
client = instructor.from_anthropic(anthropic.Anthropic())
class ExtractionResult(BaseModel):
company_name: str
founding_year: int
industry: str
result = client.messages.create(
model="claude-opus-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": "Extract: 'Stripe was founded in 2010 as a fintech company.'"}],
response_model=ExtractionResult,
)
Zod and structured output in TypeScript
TypeScript teams get the same pattern through Zod. The openai Node SDK exposes a zodResponseFormat helper that converts a Zod schema into the JSON Schema payload the API expects:
import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";
import { z } from "zod";
const client = new OpenAI();
const ExtractionResult = z.object({
companyName: z.string(),
foundingYear: z.number().int().min(1800).max(2026),
industry: z.enum(["SaaS", "Fintech", "Healthcare", "E-commerce", "Other"]),
employeeCountRange: z.enum(["1-10", "11-50", "51-200", "201-1000", "1000+"]),
});
type ExtractionResult = z.infer<typeof ExtractionResult>;
const completion = await client.beta.chat.completions.parse({
model: "gpt-4o-2024-11-20",
messages: [
{
role: "user",
content: "Extract: 'Stripe, founded in 2010, is a fintech company with 8,000+ employees.'",
},
],
response_format: zodResponseFormat(ExtractionResult, "extraction_result"),
});
const result: ExtractionResult = completion.choices[0].message.parsed!;
The parsed field is fully typed. TypeScript's type system enforces that you handle the null case (when parsing fails), which is a useful safety property at the call site.
Designing schemas that models can actually follow
Schema enforcement removes the formatting problem. It doesn't remove the semantic problem. A model can produce structurally valid JSON that's semantically wrong: a confidence score of 0.99 for a genuinely ambiguous sentiment, a founding_year of 1 for a company whose founding date wasn't in the context.
A few schema design rules that consistently improve output quality:
Use enums over freeform strings wherever possible. "sentiment": { "type": "string", "enum": ["positive", "negative", "neutral"] } is far more reliable than "sentiment": { "type": "string" }. The model's decision space is bounded and each option has a clear meaning.
Name fields descriptively. key_phrases is clearer than kp. founding_year is clearer than year. Models read field names as semantic signals; terse names produce terse reasoning about those fields.
Add description to ambiguous fields. For a field like confidence, add "description": "A score from 0.0 to 1.0 reflecting how certain the model is in its classification. Use 0.5 for genuinely ambiguous cases." This shapes the model's behavior without polluting the system prompt.
Avoid deeply nested schemas for first drafts. A flat or two-level schema is easier for the model to populate correctly than a five-level tree. If you need depth, test thoroughly, because nesting amplifies ambiguity.
Don't mark everything required. Optional fields that the model can omit cleanly produce better output than required fields the model has to hallucinate values for. Design your schema around what the model can reliably extract, not around what you'd ideally have.
The following table summarizes when to use which enforcement strategy for common AI engineering tasks:
| Task Type | Recommended Approach | Reason |
|---|---|---|
| Data extraction (documents, emails) | Constrained decoding + Pydantic/Zod | High volume, schema is fixed, failures are costly |
| Classification | Enum-constrained schema | Bounded output space, easy to validate |
| Entity recognition | Array of typed objects | Repeating structure, schema keeps count manageable |
| Summarization | Freeform or light schema | Creative task; schema hurts quality |
| Code generation | Freeform or markdown fence extraction | Schema doesn't map well to code structure |
| Agent tool calls | Tool-call schema (native API feature) | Mandatory — tool args must be parseable |
Retry, fallback, and partial-parse strategies
Even with constrained decoding, you need a retry strategy. Network errors, context-length overflows, and model refusals can all interrupt a structured output call. For schema validation paths (where you're parsing model-generated text), failures are more common.
A three-tier strategy covers most production cases:
Tier 1: Retry with the same prompt. For transient failures (network errors, rate limits), a simple retry with exponential backoff handles the majority of cases. Don't change the prompt; just retry.
Tier 2: Retry with an error-correcting prompt. For parse failures, include the failed output and the validation error in a follow-up message: "Your previous response failed validation with this error: [error]. Please correct it." This works well for Pydantic ValidationError messages, which are descriptive enough for the model to act on.
Tier 3: Partial parse and default. For non-critical fields, accept a partial result with default values rather than failing the entire request. If your schema has 10 fields and 9 parse correctly, decide whether field 10 is worth a retry. Often it isn't.
from pydantic import BaseModel, ValidationError
import time
def extract_with_retry(text: str, max_retries: int = 3) -> ExtractionResult | None:
messages = [{"role": "user", "content": f"Extract company info from: {text}"}]
for attempt in range(max_retries):
try:
completion = client.beta.chat.completions.parse(
model="gpt-4o-2024-11-20",
messages=messages,
response_format=ExtractionResult,
)
return completion.choices[0].message.parsed
except ValidationError as e:
if attempt == max_retries - 1:
return None
# Feed the error back for self-correction
messages.append({"role": "assistant", "content": str(completion.choices[0].message.content)})
messages.append({"role": "user", "content": f"Validation failed: {e}. Please fix and retry."})
time.sleep(2 ** attempt)
return None
One thing we've learned building AI automation services: retry logic should be instrumented. Log every retry attempt, the validation error that triggered it, and whether the retry succeeded. Without this data you can't tell whether a schema is genuinely hard for the model or whether you're seeing a noisy API. The observability investment pays back in schema iteration speed.
Frequently Asked Questions
Does constrained decoding reduce output quality?
It can, slightly, for complex schemas. Filtering the token vocabulary means the model's next-token probabilities are renormalized over a smaller set, which can produce less natural phrasing in free-text fields within a schema. For pure data extraction tasks, this effect is negligible. For schemas with large free-text fields (summaries, descriptions), consider whether those fields actually need to be inside the structured output or could be fetched in a separate call.
Should we use structured outputs for all LLM calls?
No. Structured outputs are the right tool when the model's response feeds directly into code: parsing, classification, extraction, agent tool calls. For conversational responses, creative tasks, or anywhere the user reads the model's output directly, freeform generation produces better results. The schema is a constraint; apply it only where the constraint serves a purpose.
What's the best library for structured outputs in Python?
For OpenAI, the native SDK's client.beta.chat.completions.parse with Pydantic is the simplest path and requires no additional dependencies. For Anthropic, instructor wraps the tool-calling pattern into a Pydantic-native API and handles retries. For multi-provider setups in our AI engineering work, instructor supports both backends with consistent syntax, which reduces the surface area of provider-switching.
How do we handle schemas that change frequently?
Version your schemas explicitly. When a schema changes in a breaking way (removing a required field, changing a field's type), treat it like an API version change. Don't modify the existing schema in place. Keep old schemas in code for as long as you might need to re-process historical data against the original contract. For schemas that evolve frequently, a schema registry (even a simple one backed by a database table) is worth building early.
Does structured output work with open-source or self-hosted models?
Yes, with caveats. Models served through vLLM, Ollama, or similar inference engines support grammar-constrained decoding via their guided_json or equivalent parameters. The reliability varies more than with hosted frontier models, and some smaller models struggle with nested schemas even with constrained decoding active. The Laxaar team's recommendation for self-hosted setups is to start with flat schemas and add nesting only after you've validated baseline reliability.
Building LLM pipelines where output reliability is blocking you? The Laxaar team works on AI engineering systems where structured outputs, observability, and eval pipelines are part of the foundation, not bolted on afterward. Reach out to discuss your architecture.
Working on something like this?
Get a fixed scope, timeline, and price within one business day — no obligation.
