# Structured Output Constraints

Forcing generation to conform to a schema through grammar-constrained decoding or validated function calls.

**Rests on:** Deterministic Validators

**Leads to:** Content Moderation Guardrails

*LLMs and Generative AI · Prompt Engineering*


## Why learn this

Structured output constraints are what let you build software that talks to an AI and trusts what comes back. Without them, a model can reply with prose when you asked for a number, add text around a JSON array, or misname a field—and your code crashes. With them, you can safely feed model output straight into a database, an API, or another system. This concept unlocks three things you will meet soon: Context Packs (where you bundle the schema you want with the prompt), Prompt Iteration (where you measure whether your constraint is holding), and AI Agents (where a model calls tools, and only a validated call can be allowed to run). If you skip this, every later lesson that depends on reliable model output will feel impossible.


## The intuition

When you ask a language model a question, it does not return a structured record. It returns a string of text—it is a text generator. So when you want 'a list of three restaurants with names and ratings', the model does not naturally give you a tidy list; it gives you sentences, maybe with bullet points, maybe with extra commentary. Structured output constraints are the set of techniques that force the model to produce text that exactly matches a pre-agreed shape, like a table with specific columns. Think of it as a form with fixed fields versus a blank sheet of paper. Without constraints, you get the blank sheet; with constraints, you get a form and the model must fill it in. The point is not to ask nicely—it is to make it structurally impossible for the model to deviate.


## An analogy

Imagine you hire an assistant to record orders at a restaurant. If you just say 'please record the orders', you might get 'table 3 wants the dosa but no onions, and they asked about the chutney'. That is useful to a human, but useless to a computer that needs a table of order, table number, and items. Now imagine you hand the assistant a pre-printed form with three boxes: TABLE NUMBER, DISH, MODIFICATIONS. The assistant is taught that the form is the only output allowed—no extra notes, no commentary, no crossing out. The form is the schema. The assistant is trained (or constrained) to only ever fill in boxes. This is exactly what grammar-constrained decoding does: it does not hope the model will follow instructions; it restricts the very act of writing so that only valid forms can be produced. Where the analogy stops working: a human can still fill a box with nonsense (write 'red' in the dish box). A grammar constraint only fixes the shape, not the content—you still need validation to check that the dish actually exists on the menu. So the constraint is about form, and validation is about content.


## Definition

Structured output constraints are techniques that force a language model to generate text that exactly matches a predefined schema, typically by restricting the allowed tokens at each step (grammar-constrained decoding) or by validating and retrying function calls, so that downstream code can parse the output without error.


## How it connects

This concept is part of Prompt Engineering — the idea that you must design not just what you ask, but how the answer is allowed to look. The library notes say it best: 'structured output should be enforced by schema, not requested politely.' You already know from your notes that few-shot examples (giving examples in the prompt) are often stronger than long instructions; that is true here too, except that a schema constraint is the strongest possible hint. This connects to Context Packs because a schema is part of the context you bundle; to Prompt Iteration because you will measure whether the constraint is holding; and to Prompt Optimization because a schema is something you can define and evaluate objectively. The book 'Large Language Models: The Hard Parts' emphasises that this is a constraint problem, not a politeness problem.


## The mathematics

The mathematics here is not calculus or algebra—it is a branch called graph theory, the study of networks of connected points. You have already used graphs without knowing it: a family tree, a map of roads, a flow chart. Here the graph is the set of all possible ways a text can be built, token by token. Each token is a word or part of a word (like 'rest' or 'aurant'). Each position in the sequence is a node in the graph, and an edge leads to the next token. At each step, the model gives a probability to every possible next token. Without constraints, it picks the most likely one. With constraints, before picking, the model is only allowed to consider the subset of tokens that could still lead to a valid completion of the schema. Imagine filling a crossword: you know the pattern of black and white squares, so at each blank you only consider letters that fit both the across and down words. The constraint is not about the meaning of the letter—it is about whether it fits the pattern. That is the whole operation: restrict the choice set at each step.

Let V be the set of all possible next tokens. Let A(x) be the set of tokens allowed by the schema at position x. The constrained probability of token t is: P_constrained(t) = P(t) / sum_{u in A(x)} P(u) if t in A(x), else 0. The model samples from this constrained distribution.


## A worked example

```
Suppose you want the model to output either the word 'yes' or the word 'no'. Your schema is a single word, chosen from {'yes', 'no'}. There is no algebra here; it is a selection problem. The graph has a start node, and from it two edges: one to 'yes', one to 'no'. After that, the text must end.

Step 1: Step 1: The model is at the start node. It assigns probabilities to each possible next token: P('yes') = 0.6, P('no') = 0.3, P('maybe') = 0.1.   # Without a constraint, the model would pick 'maybe'? No—it would pick 'yes' because that has the highest probability. But 'maybe' is possible and would be a problem.
Step 2: Step 2: Apply the constraint. The schema only allows 'yes' or 'no'. So the set of allowed tokens is {'yes', 'no'}. The model's distribution over the allowed set is now: P('yes') = 0.6 / (0.6+0.3) = 0.667, P('no') = 0.3 / (0.6+0.3) = 0.333.   # You renormalise the probabilities so they sum to 1 over the allowed set. 'Maybe' is simply not considered.
Step 3: Step 3: The model samples from the allowed set. It chooses 'yes' with 66.7% chance, 'no' with 33.3%. Suppose it picks 'yes'.   # The output is now guaranteed to be either 'yes' or 'no'—never 'maybe'.
Step 4: Step 4: The output is 'yes'. It matches the schema, so no retry is needed.   # The constraint prevented an invalid token from being generated at all.

Result: The model output is exactly one of the allowed tokens. Nothing else can appear, because the next-token selection was restricted to the valid set. This is the core of grammar-constrained decoding.
```


## From scratch, in Python

```python
# Teach the building blocks first: a list, a dictionary, a for loop.
# A list holds items in order, e.g. allowed_words = ['yes', 'no']
# A dictionary maps keys to values, e.g. probabilities = {'yes': 0.6, 'no': 0.3}
# A for loop repeats an action for each item in a collection.

def constrain(probabilities, allowed):
    # probabilities: dict of token -> probability (e.g. {'yes': 0.6, 'no': 0.3, 'maybe': 0.1})
    # allowed: list of tokens that are allowed (e.g. ['yes', 'no'])
    # Step 1: keep only the probabilities for tokens in allowed
    filtered = {}
    for token in allowed:
        if token in probabilities:  # only if the token exists in the probabilities dict
            filtered[token] = probabilities[token]
    # Step 2: sum the kept probabilities
    total = sum(filtered.values())  # e.g. 0.6 + 0.3 = 0.9
    # Step 3: renormalise each probability by dividing by the total
    constrained = {}
    for token, prob in filtered.items():
        constrained[token] = prob / total  # e.g. 0.6 / 0.9 = 0.666...
    return constrained

# Example usage
probs = {'yes': 0.6, 'no': 0.3, 'maybe': 0.1}
allowed = ['yes', 'no']
result = constrain(probs, allowed)
print(result)  # {'yes': 0.666..., 'no': 0.333...}
# Now pick the most likely allowed token:
best = max(result, key=result.get)
print(best)  # 'yes'
```


## The same thing, with a framework

```python
import json
from pydantic import BaseModel, constr, ValidationError

class Order(BaseModel):
    table_number: int
    dish: constr(min_length=1)
    modifications: str = ''

# See how this is the same idea: define the shape (schema),
# then validate the model's output against it.
# The library 'pydantic' does the validation for you,
# and frameworks like 'instructor' use pydantic to force the model
# to follow the schema by retrying if the output is invalid.
```


## What the code does

The from-scratch code shows the core arithmetic: filtering and renormalising. The framework code shows what a developer actually writes in production: you define a schema as a Python class (a pydantic model), and a library like 'instructor' or 'outlines' ensures the model's output conforms. Underneath, the library either uses grammar-constrained decoding (restricting the next-token choices) or calls the model, parses the response, and if the parse fails (via validating against the schema), it feeds the error back to the model and asks it to retry. The from-scratch part is the filtering; the framework part is the enforcement and retry loop.


## Common misconceptions

**Asking nicely is enough** — If I write 'Please return JSON' in my prompt, the model will always return valid JSON.

Models are text generators; they follow probabilities, not promises. They often add extra commentary or wrap the JSON in markdown code fences, making it invalid to parse. In fact: A prompt is a soft signal. Only a hard constraint—grammar-constrained decoding or validation with retry—guarantees the shape.

**Schema enforcement guarantees valid content** — If the model output passes schema validation, it must be factually correct.

A schema only checks shape (e.g., a string is a string), not meaning. The model can output 'table 5' for a dish name. In fact: After schema validation, you still need content validation—e.g., checking that the dish exists on the menu.

**You can always enforce any schema** — I can force the model to output any structure, no matter how complex.

Grammar constraints work on context-free grammars; they cannot express every arbitrary logic (e.g., 'the price must be less than the tax rate' is a content constraint, not a grammar rule). In fact: Formal constraints are limited to syntactic structure. Semantic rules must be checked after generation.

**Structured output is always slower** — Enforcing a schema will always slow down generation.

Constrained decoding sometimes speeds things up because the model does not waste tokens on invalid continuations, but the filtering step does add a little overhead. In fact: The net effect varies; in many cases, the reduction in retries makes it faster overall.


## In short

Structured output constraints are not about asking nicely; they are about making invalid text structurally impossible. The core idea is to restrict the model's next-token choices to those that fit a pre-defined schema, renormalising probabilities so a valid token is always chosen. This eliminates whole classes of parsing errors. Remember: the constraint handles the shape, but you still need validation to check the content. In practice, you use libraries like instructor or outlines to enforce a pydantic schema, and you always combine that with an error-handling loop for the cases where the model still produces something unusable.


## Where this is used

This concept is everywhere. In large language models (LLMs) like GPT-4, you can set 'response_format' to something like JSON to get a JSON object, but that only biases the model—it does not guarantee validity. For true guarantees, libraries like 'outlines' or 'grammar-constrained decoding' (used by Llama.cpp) restrict the token generation itself, so the output is always parseable. In small language models running on a phone or laptop (e.g., via Ollama), the same technique is used—you often do it with a grammar file (like a GBNF grammar) that specifies the allowed structure, because you cannot afford retries on a slow device. In retrieval-augmented generation (RAG), after you retrieve documents, you might want the model to produce a structured answer like a list of sources with confidence scores; structured output ensures the answer can be fed directly into a UI or a database. In agentic systems, the model must call tools like 'get_weather' or 'book_flight'. Each tool call must be a valid JSON with the right function name and arguments. A structured output constraint ensures the model only emits calls to registered tools, with correct parameter types—this is how systems like OpenAI's function calling work. If a call is malformed, the agent would crash. In everyday applied ML, consider a recommendation system that outputs a list of product IDs; you want a JSON array of integers, not prose. Fraud detection might output a decision and a confidence score; you need a number in [0,1], not a sentence. Speech systems might need to parse a date from a user's voice; the decoded text is constrained to a date format. In practice, developers reach for libraries like 'instructor', 'outlines', or 'jsonformer' to enforce schemas. They write a pydantic class, pass it to the library, and get back a validated object. What breaks when it is done badly: silent parsing failures, retries that never succeed, crashes, or worse—a system that trusts an invalid output and acts on it, like debiting the wrong account because a field was a string instead of a float.


---

Read this on the web: https://basavapurushottam.com/learn-ai/structured-output-constraints
