# Data Validation Constraints

Explicit rules a dataset must satisfy — types, ranges, uniqueness, referential integrity — enforced automatically.

**Rests on:** Data Dictionaries

**Leads to:** Constraint Generation, Roll-Up Constraints

*Data Engineering and Analytics · Data Quality*


## Why learn this

Before you can trust any AI model — a recommendation engine, a fraud detector, even a chatbot — you must trust the data it learns from. Garbage in, garbage out is not a joke; it is the single most common way AI projects fail in practice. Data validation constraints give you a way to say, 'This column must always be a number between 0 and 100' and have the computer check it automatically every time new data arrives. Mastering this unlocks a safer way of building software called Test-Driven Development for data, which is a method where you write automatic checks to verify your data’s health before you start writing the main code. Without constraints, every model you ever build is a house on sand.


## The intuition

Think of a form you have filled out online: it asks for your age, and you type 'twenty-five' instead of 25. The form rejects it — it expects a number. That is a data validation constraint in action. Now imagine you are the one designing that form. You decide: age must be a number, it must be at least 0, and it cannot be more than 120. Those three rules are constraints. In AI, the 'form' is your dataset — a big table of rows and columns. Each column has a type (numbers, text, dates), and often a range (prices are positive, probabilities are between 0 and 1). Some columns must be unique (like customer IDs), and some must refer to values in another table. For example, if you have an 'orders' table with customer names, those names must match actual customers in a separate 'customers' table; this link between tables is what we call referential integrity, meaning the data stays connected correctly and doesn't point to ghosts that don't exist. Data validation constraints are exactly those rules, but instead of a human checking each row, you write them down in code and let the computer check every single row automatically. The point is not to be annoying — it is to catch problems early, before they poison your model.


## An analogy

Consider the security checkpoint at an airport. Every passenger must show a boarding pass and a photo ID. The guard checks: is the name on the ID the same as on the boarding pass? Is the flight number correct? Is the date today? These are constraints — rules about what valid travel documents look like. If something is off, the passenger is pulled aside, not waved through. This is exactly what data validation constraints do for your dataset: they are the security guard that stops bad records before they enter your model. The analogy holds in one important way: the guard does not care whether the passenger is a good person or has a valid reason; they only check whether the documents meet the rules. Similarly, a constraint does not judge whether a data point is 'good' or 'interesting' — it only checks if the value fits the rule. The analogy breaks down, however, in that an airport guard can use judgment and context (a small discrepancy might be a typo, not a threat), while a validation constraint is rigid: a value either meets the rule or it does not.


## Definition

Data validation constraints are explicit, executable rules that a dataset must satisfy — covering data types, value ranges, uniqueness, and referential integrity (the rule that linked data must actually exist in its source table) — enforced automatically by code every time data enters a pipeline.


## How it connects

This topic sits inside a larger concept called Data Quality, which also covers Data Cleaning (fixing bad records) and Handling Missing Data (deciding what to do with empty cells). Constraints are the rules that keep data clean in the first place. They are also the foundation of Test-Driven Development for data — a way of working where you write tests (which are just constraints) before you build analysis or models.


## The mathematics

This concept does not rest on any advanced mathematics — it is closer to arithmetic and basic logic. The branch of mathematics involved is propositional logic, the study of true/false statements. That may sound abstract, but it is exactly what constraints are: each rule is a statement that is either TRUE for a record or FALSE. For example, 'Age is greater than 0' is a statement. For a record with age 25, it is TRUE; for a record with age -5, it is FALSE. The computer can check this for every row without you having to do anything manual. The mathematics here is about combining these true/false statements: you can say 'Age must be greater than 0 AND less than 120' — that is two statements joined by an AND. Both must be TRUE for the record to pass. You can also use OR, NOT, or compare values from different columns. You do not need to know algebra or calculus — you only need to think in terms of clear, black-and-white rules. But that is precisely why constraints are powerful: they reduce fuzzy 'this data looks okay' into a set of mechanical checks that never get tired or biased.

Let us look at how we write this down formally. We use the letter R to stand for one single record in your dataset D. Think of D as the whole table, and R as just one row in that table.

We define a rule, which we call c (read 'c'). This rule takes the record R as input and gives back a simple answer: TRUE or FALSE. We write this as c(R). If the record follows the rule, c(R) is TRUE. If it breaks the rule, c(R) is FALSE.

Suppose you have k rules in total. We label them c1, c2, up to ck. The letter k just means 'the total number of rules'. For a record to be valid, it must pass every single one of these rules. In logic, we join the results with the word AND. So the condition for a valid record is:
c1(R) AND c2(R) AND ... AND ck(R)

This entire chain of ANDs must evaluate to TRUE. If even one part is FALSE, the whole thing is FALSE.

The dataset passes validation if this condition holds for every record R in D. In other words, there is no record in your table that makes any rule return FALSE.


## A worked example

We have a small dataset of customer records. Each record has: customer_id (a number), age (a number), email (text), and product_id (a number). We want to enforce the following constraints: customer_id must be unique; age must be between 0 and 120; email must contain '@'; product_id must be one of the valid products (1001, 1002, 1003). Here are three sample records:

Record 1: customer_id=1, age=25, email='alice@example.com', product_id=1001
Record 2: customer_id=2, age=-3, email='bob@example.com', product_id=1002
Record 3: customer_id=1, age=30, email='charlieexample.com', product_id=9999

Let us define our rules explicitly so we can check them step by step.
Rule 1 (c1): Is the customer_id unique? This requires looking at all previous records. If this is the first time we see this ID, it is TRUE. If we have seen it before, it is FALSE.
Rule 2 (c2): Is age between 0 and 120? Mathematically, this means: age >= 0 AND age <= 120.
Rule 3 (c3): Does the email contain the symbol '@'?
Rule 4 (c4): Is product_id in the set {1001, 1002, 1003}?

Now we apply these rules to each record.

Step 1: Process Record 1.
Record 1 data: customer_id=1, age=25, email='alice@example.com', product_id=1001.

Check Rule 1 (c1): Is customer_id=1 unique? This is the first record. We have not seen ID 1 before. So c1(Record 1) is TRUE.

Check Rule 2 (c2): Is age=25 between 0 and 120?
Is 25 >= 0? Yes.
Is 25 <= 120? Yes.
Both parts are true, so c2(Record 1) is TRUE.

Check Rule 3 (c3): Does 'alice@example.com' contain '@'? Yes. So c3(Record 1) is TRUE.

Check Rule 4 (c4): Is product_id=1001 in {1001, 1002, 1003}? Yes. So c4(Record 1) is TRUE.

Combine the results for Record 1:
c1(Record 1) AND c2(Record 1) AND c3(Record 1) AND c4(Record 1)
= TRUE AND TRUE AND TRUE AND TRUE
= TRUE.
Record 1 passes all constraints.

Step 2: Process Record 2.
Record 2 data: customer_id=2, age=-3, email='bob@example.com', product_id=1002.

Check Rule 1 (c1): Is customer_id=2 unique? We have not seen ID 2 before. So c1(Record 2) is TRUE.

Check Rule 2 (c2): Is age=-3 between 0 and 120?
Is -3 >= 0? No.
Since the first part is false, the whole condition fails immediately. c2(Record 2) is FALSE.

Check Rule 3 (c3): Does 'bob@example.com' contain '@'? Yes. So c3(Record 2) is TRUE.

Check Rule 4 (c4): Is product_id=1002 in {1001, 1002, 1003}? Yes. So c4(Record 2) is TRUE.

Combine the results for Record 2:
c1(Record 2) AND c2(Record 2) AND c3(Record 2) AND c4(Record 2)
= TRUE AND FALSE AND TRUE AND TRUE
= FALSE.
Record 2 fails because Rule 2 is false. The age is negative, which is not allowed.

Step 3: Process Record 3.
Record 3 data: customer_id=1, age=30, email='charlieexample.com', product_id=9999.

Check Rule 1 (c1): Is customer_id=1 unique? We saw ID 1 in Record 1. So this is a duplicate. c1(Record 3) is FALSE.

Check Rule 2 (c2): Is age=30 between 0 and 120?
Is 30 >= 0? Yes.
Is 30 <= 120? Yes.
c2(Record 3) is TRUE.

Check Rule 3 (c3): Does 'charlieexample.com' contain '@'? No. c3(Record 3) is FALSE.

Check Rule 4 (c4): Is product_id=9999 in {1001, 1002, 1003}? No. c4(Record 3) is FALSE.

Combine the results for Record 3:
c1(Record 3) AND c2(Record 3) AND c3(Record 3) AND c4(Record 3)
= FALSE AND TRUE AND FALSE AND FALSE
= FALSE.
Record 3 fails on three counts: duplicate ID, invalid email format, and unknown product ID.

Result: Records 2 and 3 fail one or more constraints. Record 1 passes all. If this were a production pipeline, records 2 and 3 would be blocked (or flagged) before they reach the model — preventing bad data from silently corrupting your analysis.


## From scratch, in Python

```python
# We'll use a list of dictionaries to represent our dataset.
# A dictionary looks up values by name, like a phone book.
# A list keeps things in order.

data = [
    {'customer_id': 1, 'age': 25, 'email': 'alice@example.com', 'product_id': 1001},
    {'customer_id': 2, 'age': -3, 'email': 'bob@example.com', 'product_id': 1002},
    {'customer_id': 1, 'age': 30, 'email': 'charlieexample.com', 'product_id': 9999},
]

# Valid product IDs (like a mini reference table)
valid_products = {1001, 1002, 1003}

# We'll collect errors as a list of strings
errors = []

# Keep track of seen customer IDs to check uniqueness
seen_ids = set()

for row in data:  # 'for each row in the list'
    # Type check: age must be an integer
    if not isinstance(row['age'], int):
        errors.append(f"Row {row['customer_id']}: age is not an integer")

    # Range check: age between 0 and 120
    if row['age'] < 0 or row['age'] > 120:
        errors.append(f"Row {row['customer_id']}: age {row['age']} out of range")

    # Uniqueness check: customer_id must not be seen before
    if row['customer_id'] in seen_ids:
        errors.append(f"Row {row['customer_id']}: duplicate customer_id")
    seen_ids.add(row['customer_id'])

    # Format check: email must contain '@'
    if '@' not in row['email']:
        errors.append(f"Row {row['customer_id']}: email missing '@'")

    # Referential integrity: product_id must exist in valid_products
    if row['product_id'] not in valid_products:
        errors.append(f"Row {row['customer_id']}: unknown product_id {row['product_id']}")

# Print all errors, or success if none
if errors:
    for e in errors:
        print(e)
else:
    print("All records pass validation.")
```


## The same thing, with a framework

```python
In real data engineering, tools like Pandas have built-in validation functions, but the concept is the same. For example, `df['age'].between(0, 120)` returns a boolean Series. You can use it to filter violations. Validation libraries like Great Expectations or pandera let you declare schemas: you say 'column age must be int and between 0 and 120' in one line, and they check it for you. Under the hood, they are doing exactly the loop we wrote above — iterating over values and evaluating rules.
```


## What the code does

We start by defining `data` as a list of dictionaries. Each dictionary is one record, with column names as keys. This is a common way to store tabular data in Python. Next, we have `valid_products` — a set (a collection that holds each value once, no order). We need it to check referential integrity. Then we create an empty `errors` list and an empty `seen_ids` set. The `for row in data:` loop goes through each record one at a time. For each record, we check several conditions. `isinstance(row['age'], int)` verifies the value is an integer type — this catches strings or floats. `if row['age'] < 0 or row['age'] > 120:` is a range check; `or` means if either side is TRUE, the whole condition is TRUE. Uniqueness: we check if the ID is already in `seen_ids`; if not, we add it. For email, `if '@' not in row['email']` checks the substring. Finally, `if row['product_id'] not in valid_products` checks membership in the set. Each failure appends a message to `errors`. After the loop, we print all errors or a success message. This is the heart of validation: a list of checks, each producing a TRUE/FALSE answer.


## Common misconceptions

**Constraints are optional if the data is clean** — My data comes from a trusted source, so it will always be clean. I don't need to write validation rules.

Even trustworthy sources change over time: a developer may change a format, a system may start sending nulls, and eventually something will break. The cost of catching a problem early is tiny compared to the cost of finding it after a model has been trained and deployed. In fact: Constraints are not a commentary on data quality; they are a safety net. Always enforce them, even on clean data, because data changes.

**A constraint failure means the data is bad and must be deleted** — If a record fails a constraint, I should just drop it from the dataset.

Sometimes the rule itself is wrong (e.g., age > 0 might reject a legitimate age of 0 for a newborn), or the failure is a signal worth investigating. Deleting records without review can introduce bias or hide a systemic issue. In fact: Failures should be investigated, not automatically discarded. The constraint flags the record; the analyst decides what to do. Sometimes the rule needs to change, sometimes the data does.

**Constraints and data cleaning are the same thing** — If I have constraints, I don't need to clean data separately.

Constraints detect problems; cleaning fixes them. A constraint can say 'age must be positive', but it doesn't tell you whether to change -5 to 5, 28, or drop the record. Cleaning is a separate decision. In fact: Constraints are the first line of defense — they tell you *that* something is wrong. Cleaning is the second step — it decides *what to do* about it.

**Bias in data and bias in constraints are the same thing** — When people talk about 'bias' in AI models, they mean the bias parameter in a neural network. Validation constraints are unrelated.

Actually, they are related: if your constraints are biased (e.g., they exclude certain groups because a threshold is too strict), they will produce a biased dataset, and the model will inherit that bias. But the word 'bias' has multiple meanings: a model parameter, a systematic error, and a statistical term. In fact: Constraints are rules; if any group of records disproportionately fails, the rules may be systematically unfair. Always check that your constraints are not inadvertently excluding certain categories of data.


## In short

Data validation constraints are explicit rules that keep your data honest. They check types, ranges, uniqueness, and referential integrity — automatically, every time. They are not a formality; they are the gate that stops bad data from reaching your model. Failures should block the pipeline, not just print a warning. Constraints also document what the data is supposed to look like, for anyone who reads your code later. Write them before you trust any dataset, and let the computer check every row, every time.


## Where this is used

Data validation constraints are used in every AI system, not as a model component but as the gatekeeper of data quality.

**In large language models (LLMs):** Training data for models like GPT must be clean and well-formatted. Constraints are applied to ensure text comes from allowed sources, that each document has a minimum length, that there are no duplicate passages (which would bias the model), and that content passes safety filters (e.g., no toxic text). At inference time, when users input prompts, constraints check that the input is text, not empty, and within a length limit, before it is passed to the model. A developer writes these as validation functions that run before the model call.

**In small models running on devices (TinyML):** A smart speaker that wakes on 'Hey device' runs a tiny model on-device. The raw audio input must meet constraints: sample rate 16 kHz, duration 0.5–2 seconds, amplitude within a range. If a silent recording arrives, constraints reject it and the model never runs — saving battery. A developer writes a preprocessing script that checks these properties before feeding the audio to the model.

**In retrieval-augmented generation (RAG):** RAG systems fetch relevant documents to help an LLM answer. The document store must satisfy constraints: every chunk has a non-empty text, a unique ID, and a valid embedding vector of the expected dimension. If a chunk is missing an embedding, retrieval breaks. Constraints check this when documents are ingested. A developer writes a validation that checks embedding length = 768 (for a common model), and if not, re-embeds or rejects the chunk.

**In agentic systems (AI agents that plan and call tools):** Agents make decisions and take actions, such as booking a flight or sending an email. Validation constraints on the agent's outputs are critical for safety: the date must be in the future, the email address must be valid, the amount must be positive. Without these, an agent could send a booking for a past date or transfer a negative amount. In practice, developers wrap agent tool calls with validators that check arguments before execution. For example, a tool call `book_flight(date='2024-01-01')` would be rejected because the date has passed.

**In everyday applied ML (recommendation, vision, forecasting, fraud, speech):** Consider a fraud detection model: training data must have labels 0 or 1 (binary), transaction amounts must be positive, and customer IDs must match a reference table. If a negative amount appears, the model learns nonsense. In forecasting, timestamps must be sequential and complete; constraints catch missing dates before training. In computer vision, image dimensions and color channels must be consistent; a validation constraint rejects a 224x224 RGB image that arrives as 224x224 grayscale, which would break the model. In speech recognition, audio files must be in the expected format (e.g., WAV, 16 kHz, mono); constraints ensure that.

**What a developer actually writes:** They use Python libraries like Pydantic, Great Expectations, or custom assertion functions. For each dataset, they write a schema: a list of constraints on each column. When new data arrives, they run `validate(data)` which returns a report of violations. If there are any, the pipeline stops (or alerts) rather than silently proceeding.

**What breaks when done badly:** If constraints are too loose (e.g., 'age is a number'), bad records with age 999 slip through, and the model's predictions become unreliable. If constraints are too strict (e.g., 'age must be 18-65'), legitimate records are rejected, and the model sees a biased sample. The worst case is silent failure: a constraint is not enforced (or not written), and the pipeline continues, producing wrong outputs that no one notices until the model performs terribly in production.


---

Read this on the web: https://basavapurushottam.com/learn-ai/data-validation-constraints
