# Deterministic Validators

Programmatic checks — schema validity, format, forbidden content, numerical bounds — that give unambiguous pass or fail signals.

**Rests on:** Evaluation Datasets

**Leads to:** Content Moderation Guardrails, Structured Output Constraints

*LLMs and Generative AI · LLM Evaluation*


## Why learn this

This concept gives you the first, most reliable layer of checking any AI system's output. Before you spend money on a model judge or hours reading human review, deterministic validators catch the failures that are unambiguous: the answer isn't in the right format, contains forbidden content, or is numerically out of range. They are the cheapest, fastest, and most reproducible checks you will use. This unlocks further study of model-based evaluation — knowing when a deterministic check is enough and when you need an LLM-as-judge. Without this foundation, you will later be tempted to use an expensive model judge for something a simple rule could catch, wasting time and money.


## The intuition

Imagine you are checking a form someone filled in by hand. You have a list of rules: the date must be written as day/month/year, the phone number must have exactly 10 digits and start with 9, 8, 7, or 6, the 'age' field must be a number between 0 and 120, and the 'comments' box must not contain the word 'bomb'. None of these rules require judgment or interpretation. A date like '32/13/2024' is just wrong — the day can't be 32. A phone number like '123' is too short. An age of 200 is impossible. And if someone writes 'bomb' in the comments, you reject it without reading the rest. You do not need a brilliant assistant to check these; a simple clerk with a checklist can do it. In AI, a deterministic validator is exactly that checklist. It takes the output of a model — text, a JSON object, a number — and applies rules that have only two possible results: PASS or FAIL. There is no 'maybe'. This is in contrast to a model-based judge, where a large language model looks at an answer and decides, with some probability, if it is good. Deterministic validators do not guess; they follow rules. In the world of AI, where models can produce almost any output, having a set of checks that are unambiguous is like having a solid foundation for a building — you can build fancier things on top, but you need this base first.


## An analogy

Think of a security guard at a gated entrance. His job is to check everyone who wants to enter. He has a clipboard with a list of checks: Does the person have a valid ID? Is the ID's photo matching their face? Is the ID's expiry date in the future? Is the person on the do-not-enter list? These checks are binary — yes or no. He does not evaluate the person's character or whether they seem nice; he just checks the facts. If any check fails, the person is stopped. A deterministic validator is that security guard. It does not use a magic crystal ball; it uses a checklist of rules that are written in code. Now, consider where this guard's power ends. He cannot decide if a person is legally allowed to enter a country — that needs an immigration officer with judgment. He cannot assess if a person is a security risk based on their behaviour — that needs expert analysis. The guard's checks are only as good as the checklist. If the checklist is incomplete, a clever person might get through (in AI, a model might produce a subtly harmful answer that passes all format checks). So deterministic validators are powerful for what they are, but they are only a first line, not a complete solution. They are cheap, fast, and never get tired, but they cannot judge quality or intent.


## Definition

A deterministic validator is a programmatic check — such as schema validation, format matching, forbidden content scanning, or numerical bound testing — that returns an unambiguous PASS or FAIL signal based on a fixed set of rules, with no randomness and no model-based judgment.


## How it connects

You may have heard of the 'Evals Gap' — the widespread failure to properly evaluate AI applications before shipping. Deterministic validators are the first step in closing that gap. They are the cheapest and most reliable evaluation tool in your arsenal. Also, they connect to 'Evaluation Datasets': you will build a set of inputs to test your validators, to ensure they catch what they should. And when you later learn about 'LLM as Judge', you will see that deterministic validators are the layer that should run before a model judge, catching the easy structural failures so the judge only handles the hard cases. They also relate to 'Answer Relevancy and Correctness' — a deterministic validator can check for a format that implies relevance or a fact that implies correctness, but it cannot judge whether the answer truly addresses the question. You have not yet studied any of these topics, but this concept will be the bedrock you build on.


## The mathematics

The mathematics behind deterministic validators is not calculus or algebra — it is the most basic branch: arithmetic. You are comparing numbers: is this value less than 10? Is this value greater than 0? Is this string exactly 10 characters long? You might also have to join multiple checks with logical 'and' or 'or'. For example, you might say: PASS if the age is between 0 and 120 AND the phone number has 10 digits. This is a single rule that is a combination of two smaller rules. When you build a validator, you are essentially defining a function that takes an input (like a piece of text or a JSON object) and applies these arithmetic and logical operations to produce a binary output: TRUE (PASS) or FALSE (FAIL). The mathematics is simple, but the logic can be intricate, which is why we write it in code rather than doing it by hand. The branch of mathematics is discrete logic — the study of true/false statements — but the operations themselves are the arithmetic you learned in school: comparisons like 'less than', 'greater than', and 'equals'. There is no continuous change here; everything is a yes/no decision.

Let a be the age (a number) and let valid = (a >= 0) and (a <= 120).

The validator returns TRUE (PASS) if valid is TRUE, otherwise FALSE (FAIL).

Symbol glossary: a = the value being checked, >= means 'is greater than or equal to', <= means 'is less than or equal to', and is a logical operator that means 'both conditions must be true'.


## A worked example

```
Let's say we have a function that takes a person's age as a number and wants to check if it is a valid age. The rule is: the age must be at least 0 and at most 120. We will test a few ages: 25, -5, 150.

Step 1: Check if 25 >= 0 → TRUE
Step 2: Check if 25 <= 120 → TRUE
Step 3: Combine with AND: TRUE AND TRUE → PASS   # Both checks passed, so the overall result is PASS.
Step 4: Check if -5 >= 0 → FALSE (because -5 is less than 0)
Step 5: Combine with AND: FALSE AND … → FAIL (no need to check the second condition, because first failed)   # When one condition fails, the whole rule fails. This is called short-circuiting.
Step 6: Check if 150 >= 0 → TRUE
Step 7: Check if 150 <= 120 → FALSE (because 150 is greater than 120)
Step 8: Combine with AND: TRUE AND FALSE → FAIL   # Second condition failed, so overall fail.

Result: The validator returns PASS for an age of 25, and FAIL for -5 and 150. This is unambiguous: any age outside the range 0-120 is rejected.
```


## From scratch, in Python

Let's write a validator in plain Python. First, we need a function that checks if a number is a valid age. We'll use a function, which is a block of code that takes an input and returns an output.

```python
def is_valid_age(age):
    if age >= 0 and age <= 120:
        return True
    else:
        return False
```

Now, let's use it:
```python
print(is_valid_age(25))   # True
print(is_valid_age(-5))   # False
print(is_valid_age(150))  # False
```

Now, let's do a more realistic check: a text output from an AI should be in JSON format. We'll use a dictionary (a data structure that stores key-value pairs) to represent the expected schema. We'll check that the 'type' is 'report' and the 'score' is a number between 0 and 100.

```python
def validate_report(output):
    # output is a dictionary from the AI
    if 'type' not in output or 'score' not in output:
        return False
    if output['type'] != 'report':
        return False
    score = output['score']
    if not isinstance(score, (int, float)):  # is it a number?
        return False
    if score < 0 or score > 100:
        return False
    return True
```

Let's test it:
```python
print(validate_report({'type': 'report', 'score': 85}))   # True
print(validate_report({'type': 'summary', 'score': 85}))  # False, wrong type
print(validate_report({'type': 'report', 'score': 150}))  # False, score too high
```

Now, let's check for forbidden content. Suppose we don't want the output to contain the word 'error'. We'll use `in` to check if a substring is present.

```python
def no_forbidden_words(text):
    if 'error' in text.lower():
        return False
    return True
```

Testing:
```python
print(no_forbidden_words('The process completed successfully.'))  # True
print(no_forbidden_words('There was an error in the process.'))   # False
```

We can combine these into a single validator function that runs all checks and returns True only if all pass.

```python
def validate_full_output(output):
    if not validate_report(output):
        return False
    # assume output['text'] is the full text
    if not no_forbidden_words(output.get('text', '')):
        return False
    return True
```


## The same thing, with a framework

In practice, you rarely write your own validator from scratch — you use a validation library. Pydantic is the standard for Python. It lets you define a schema as a class, and it automatically validates data. For example:

```python
from pydantic import BaseModel, Field

class Report(BaseModel):
    type: str = 'report'
    score: float = Field(ge=0, le=100)  # must be between 0 and 100

# This will throw a validation error if invalid
Report(type='report', score=85)  # works
Report(type='report', score=150) # raises ValidationError
```

Under the hood, Pydantic is doing exactly what we did by hand — checking the type, the bounds, and returning a pass/fail signal (or raising an error). The benefit is that it is concise and battle-tested, so you don't have to write the checks yourself, and it handles many edge cases (like strings that can be converted to numbers).


## What the code does

The first function `is_valid_age` takes a number `age` and uses an `if` statement to check two conditions with `and`. If both are true, it returns `True`, otherwise `False`. The second function `validate_report` uses a dictionary `output`. Each check is a separate `if` that returns `False` if the check fails. Note the use of `not in` to check that a key exists. The third function checks for the substring 'error' in the text, converting to lowercase to catch 'Error' and 'ERROR'. The final function combines the checks, calling the previous functions. If any returns `False`, the whole thing returns `False`.


## Common misconceptions

**Deterministic validators are only for data entry** — This is just for forms on a website, not for AI.

They are used throughout AI systems to validate model inputs and outputs, ensure function calls are correct, and enforce safety rules. In fact: Deterministic validators are essential in AI engineering — they validate the structure of LLM outputs, check tool arguments, and enforce business rules before more expensive checks run.

**A deterministic validator can judge answer quality** — If a validator passes, the answer must be good.

Validators only check objective properties like format and bounds; they cannot understand meaning. A perfectly formatted answer can be factually wrong. In fact: Deterministic validators are the first gate but not the whole evaluation. They ensure structural correctness; quality needs model-based judges or human review.

**Validators are expensive to write and maintain** — It takes a lot of time and effort to build validators.

Using libraries like Pydantic, you can define a schema in a few lines. Maintenance is low because rules are simple and explicit. In fact: Modern libraries make validators concise. The cost is minimal compared to the time saved by catching errors early.

**A validator using an LLM is deterministic** — If I ask an LLM 'is this valid?', that's a deterministic validator.

LLMs are stochastic — same input can give different outputs. Deterministic validators follow fixed rules, so they are reproducible. In fact: Deterministic validators are rule-based, not model-based. If a check uses a language model, it is a model-based judge, which is non-deterministic.

**One validator is enough** — I can write one big check that covers everything.

Different checks have different failure modes, and mixing them makes it hard to know what failed. Modular validators are easier to test and maintain. In fact: Break validators into small, focused functions. Combine them in a pipeline, so you know exactly which check failed.

**Validators only work on simple data like numbers** — You can't validate text or complex JSON.

You can validate strings (length, regex), lists (length, item types), JSON structures (keys and types), and more. In fact: Validators work on any structured data: strings, numbers, lists, dictionaries, and even combinations. Libraries like Pydantic handle nested schemas.


## In short

Deterministic validators are simple rule-based checks that produce an unambiguous PASS or FAIL. They are cheap, fast, and fully reproducible. They are used everywhere in AI to validate model outputs, check tool calls, enforce bounds, and catch structural errors before they cause problems. They are not a replacement for human judgment or model-based evaluation, but they are the first, essential layer of any robust AI system.


## Where this is used

Where are deterministic validators used in the real world? Almost everywhere in AI systems. Let's go through the list.

**Large Language Models (LLMs)**: When you have an LLM answer a query, you often ask it to return JSON (a structured data format). But LLMs sometimes forget the closing brace, or add extra text. A deterministic validator checks that the output is *valid JSON* — that it can be parsed, that it has the expected keys and value types. For example, a financial assistant might be required to output `{'ticker': 'AAPL', 'price': 198.5}`. The validator checks: is it valid JSON? Does it have 'ticker' and 'price'? Is 'price' a number? This is often done with a library like Pydantic, which you can call to parse the LLM's output. If it fails, you can retry the LLM with a prompt like 'Your response must be valid JSON' — a pattern called 'retry with correction'.

**Small Language Models on devices (like phones)**: On-device models that do things like autocomplete or voice assistants can output text. A deterministic validator might check that the response is within a maximum length (e.g., 50 characters) to avoid overflowing the screen, or that it does not contain profanity. These checks are cheap and run locally without network calls, which is crucial for privacy and speed.

**Retrieval-Augmented Generation (RAG)**: In RAG, a system retrieves documents and asks an LLM to answer based on them. A deterministic validator can check that the retrieved documents are not empty or that the final answer does not say 'I don't know' when the source documents actually contain relevant information (though that might be a nuance). More importantly, the validator checks that the answer includes a citation in the correct format, e.g., '[1]' at the end. If not, the system can re-prompt or flag the answer as invalid.

**Agentic Systems (agents that plan and call tools)**: This is where deterministic validators shine. An agent decides to call a function, e.g., `get_weather(city='Mumbai')`. The agent's output is a structured call like `{'function': 'get_weather', 'parameters': {'city': 'Mumbai'}}`. A deterministic validator checks that the function name is in the allowed set, that the 'parameters' object has a 'city' key, and that 'city' is a string. If the agent tries to call a function that doesn't exist, or passes a number where a string is expected, the validator fails it. This prevents runtime crashes and is a core part of reliable agent frameworks. In code, you would define an allowed functions list and validate against it.

**Everyday Applied ML (recommendation, vision, fraud detection, speech)**: 
- **Recommendation systems**: When a model returns a list of recommended products, a validator checks that the list is not empty, has a maximum length, and every item is a valid product ID. If a product is out of stock, a deterministic check can filter it out.
- **Vision**: For an object detection model, a validator checks that the bounding box coordinates are within the image boundaries (e.g., x is between 0 and image width). It also checks that the confidence score is between 0 and 1.
- **Fraud detection**: A model predicts a probability of fraud (e.g., 0.85). A deterministic validator checks that the probability is in [0,1] and that the decision threshold was applied correctly. It also checks that the input features were in the expected range (e.g., transaction amount not negative).
- **Speech**: For a speech-to-text system, a validator checks that the output text is non-empty, below a max length, and does not contain all repeated characters (e.g., 'aaaaaaaa').

**What a developer writes**: In practice, they use a library like Pydantic (Python) to define a model of the expected output, and then they call `Model.parse_obj(llm_output)`. If it fails, they catch the exception and handle it — by retrying, logging, or flagging. They also write tests for their validators (using a framework like pytest) to ensure the validator correctly accepts valid inputs and rejects invalid ones.

**What breaks when done badly**: If you skip deterministic validators, your system will fail in subtle ways. An LLM might return malformed JSON, and your code will crash with an obscure error. A tool-calling agent might call a nonexistent function, causing a runtime error. A recommendation model might return an empty list, showing no recommendations to users. And if you don't validate inputs, your model could receive out-of-range values (like a negative age) and produce nonsense. Deterministic validators are the first line of defense; skipping them is like building a house without a foundation.


---

Read this on the web: https://basavapurushottam.com/learn-ai/deterministic-validators
