# Data Contracts

Explicit agreements between data producers and consumers covering schema, semantics, quality and change process.

**Rests on:** Semantic Layer

**Leads to:** Parquet and Flat-File Hygiene

*Data Engineering and Analytics · Data Engineering*


## Why learn this

Imagine you are working on a project where one team sends data to another team's dashboard or automated report. Without a clear agreement on what that data contains, the receiving team constantly has to guess. They might assume a number is there when it is actually missing, or interpret a date format differently. This leads to broken reports and mistrust between teams. A data contract removes this guessing game. It ensures that the team sending the information and the team using it share the same understanding of what the data means and how it looks. This allows you to change one part of your system without accidentally breaking everyone else's work, saving time and preventing costly errors in downstream processes like automated machine learning models or large-scale data processing.


## The intuition

Consider a simple agreement between two people sharing a recipe book. Before cooking, you must agree on the meaning of specific terms. For instance, when a recipe says 'flour', it clearly means wheat flour, not besan. When it says '1 cup', it refers to the same standard measuring cup for both of you. If these agreements do not exist, your friend might add sugar when you meant salt, or use a different-sized cup, and the resulting dish will be wrong. A data contract functions similarly but between computer systems. One system (the producer) sends data, such as information from a sensor or a database table. Another system (the consumer), like a dashboard or an automated model, receives and uses that data. The contract explicitly states what the data looks like (the structure of fields and their types), what each field means (the specific definition or rule for that data), how accurate it must be, and how changes to this structure are handled. Without this agreement, the consumer might assume a column contains numbers but receive text instead, or assume a value is always positive when it can be negative. This results in incorrect answers from models or reports. The contract ensures both sides agree on these details before any data is processed.


## An analogy

Think of a bus timetable shared between the bus company and a passenger. The bus company promises that a specific bus will arrive at 8:05 AM at this stop, going to the city centre. The passenger relies on this precise promise to catch their connection to work. If the bus arrives at 8:04, the passenger might miss it; if it arrives at 8:06, they are late. The timetable acts as a contract: it defines the exact arrival time (the structure of the data), the specific route and destination (the meaning of the data), the reliability standard (how often the bus operates on time), and the process for updates (new timetables are published weeks in advance, not changed silently). This analogy shows how contracts set expectations and reduce surprise. However, there is a key difference: a bus timetable is read by humans who can tolerate small ambiguities or wait if the bus is slightly late. A data contract is read by computer systems, which follow rules literally. If a system expects a number but receives text, it will either crash or produce garbage results immediately. Therefore, the contract must be precise and understandable to machines, not just people. Additionally, while bus timetables remain static for months, data structures may change frequently as new features are added, requiring a clear protocol for how both sides adapt without breaking.


## Definition

A formal agreement between a system that generates data and a system that uses it, which clearly defines the structure of the data fields, the specific meaning of each field, the required quality standards, and the rules for how changes to this structure will be managed and communicated.


## How it connects

This concept is the starting point for understanding how different parts of a data system communicate. It directly relates to the 'Semantic Layer', which is the component that defines exactly what business metrics mean by providing a shared understanding of data definitions across an organization. It also connects to 'Data Fabric and Mesh', architectures that treat data as a product owned by specific teams, where the interface between these teams is defined by such contracts. Finally, it relates to how data is stored in efficient formats like Parquet files, which embed structural information about the data columns directly within the file to help systems read and process the information correctly.


## The mathematics

This concept has no heavy math, but it leans on one everyday mathematical idea: a *set* and a *rule for comparing*. In arithmetic, you compare numbers: 5 is greater than 3, 2 equals 2. A data contract is a rule for comparing the actual data against the promised data. For each field, you check: does the value have the right type (is it a number, not text)? Does it fall in the allowed range (is it between 0 and 100, not -5)? Is it present (not missing)? These are all arithmetic or logical comparisons: equals, less than, greater than, is-not-null. The branch of mathematics is *logic and set theory* rather than calculus, because it's about categories and conditions, not about smooth change. The operation is the comparison: you produce a 'true' or 'false' for each check, and the contract says all checks must be true. There's no formula to solve, just a list of checks to perform.

For each record R with fields f₁, f₂, ..., fₙ, the contract C is a set of checks c₁, c₂, ..., cₘ. The record is valid if and only if all checks pass: valid(R) = (c₁(R) AND c₂(R) AND ... AND cₘ(R) == TRUE). Each check is a comparison like fᵢ <= max_i or fᵢ IS NOT NULL.


## A worked example

```
A bicycle rental service sends data to a pricing model. The contract says each record must have: 'bike_id' as a positive integer, 'duration_minutes' as a number between 1 and 1440, 'member' as either 'yes' or 'no'. The producer sends three records. We check each against the contract.

Step 1: Record 1: {bike_id: 42, duration_minutes: 30, member: 'yes'}   # Check: bike_id 42 is an integer and positive (42 > 0) → true. Duration 30 is between 1 and 1440 → true. Member 'yes' is in the allowed list → true. All checks pass: record is valid.
Step 2: Record 2: {bike_id: -3, duration_minutes: 0, member: 'maybe'}   # Check: bike_id -3 is not positive (negative) → false. Duration 0 is not >= 1 → false. Member 'maybe' is not in the allowed list → false. The record fails the contract.
Step 3: Record 3: {bike_id: 'abc', duration_minutes: 2000, member: 'yes'}   # Check: bike_id 'abc' is not an integer (it's text) → false. Duration 2000 is not <= 1440 → false. Member 'yes' passes, but the other two fail, so the whole record is rejected.

Result: Only record 1 conforms to the contract. Records 2 and 3 are invalid and should be rejected or flagged. The contract defines what 'valid' means, and the check is a series of comparisons, each returning true or false.
```


## From scratch, in Python

Here's a plain-Python way to check a record against a contract. We'll use a dictionary to hold a record (a dictionary maps names to values, like a phonebook maps names to numbers).

```python
# A record: a dictionary mapping field names to values
record = {'bike_id': 42, 'duration_minutes': 30, 'member': 'yes'}

# The contract: a list of checks. Each check is a function that takes a record and returns True or False.
# A function is a named block of code you can run later.

def check_bike_id_positive(record):
    # record['bike_id'] gets the value for the key 'bike_id'
    return record['bike_id'] > 0

def check_duration_range(record):
    return 1 <= record['duration_minutes'] <= 1440

def check_member_allowed(record):
    return record['member'] in ['yes', 'no']  # 'in' checks if the value is in the list

# Put all checks in a list
checks = [check_bike_id_positive, check_duration_range, check_member_allowed]

# A loop: for each check in the list, run it and if it fails, reject the record
valid = True
for check in checks:
    if not check(record):
        valid = False
        print(f"Failed check: {check.__name__}")
        break  # stop at first failure

if valid:
    print("Record is valid")
else:
    print("Record is invalid")
```

Now test with a bad record:

```python
bad_record = {'bike_id': -3, 'duration_minutes': 0, 'member': 'maybe'}
# Run the same loop with bad_record instead of record
# Output: Failed check: check_bike_id_positive
# Output: Record is invalid
```


## What the code does

First, we make a dictionary called 'record' — it holds named values, so you can look up 'bike_id' and get 42. Then we define three functions. Each function takes a record and returns True if that record passes that one rule. For example, check_bike_id_positive checks that the bike_id is greater than 0. We put all three functions in a list called 'checks'. Lists keep things in order and can be looped over — here, we loop over the list. For each check, we call check(record), which runs the function. If it returns False, we set valid to False, print which check failed (using __name__ to get the function's name), and stop (break). If no check fails, valid stays True. This is a simple example: a real contract validation would have dozens of checks and read from a file, but the idea is the same: a list of rules, loop over them, flag failures.


## Common misconceptions

**A data contract is just a document.** — I'll write a Google Doc explaining the fields and send it to the team. That's a contract.

A document is read by humans and can be interpreted differently. A data contract is enforced by code — it's a machine-readable schema and validation rules that automatically reject data that doesn't conform. A human might miss a change; code will not. In fact: A data contract is a machine-readable specification (like JSON Schema) plus automated validation. It's not a suggestion; it's an invariant enforced by the system.

**Data contracts are only for APIs.** — We're not building an API, we just have a database. We don't need a contract.

Any producer–consumer relationship needs a contract. A database table is produced by one application and consumed by another. The database schema is a weak contract — it doesn't enforce semantics (e.g., 'status' could mean three different things) or quality (e.g., completeness). Without a contract, a change to the table definition breaks every query. In fact: Data contracts apply to databases, files, message queues, and any data exchange, not just API calls. They define the rules above and beyond what the database schema enforces.

**A contract that is too strict will block everything.** — If I require that 'duration' is always between 1 and 1440, I'll reject valid data if someone records a duration of 0 (a free rental). The contract is too rigid.

The contract should reflect the actual domain rules. If duration 0 is valid, the contract should say so — the range should be '>= 0'. The issue isn't the strictness; it's that the contract is inaccurate. The contract is a negotiation: if the producer genuinely needs to send 0, they can request a change to the contract. In fact: A good contract is accurate, not necessarily strict. It's updated through a change process to reflect real requirements, and it's enforced so that invalid data is caught, not just recorded.

**Data contracts slow us down.** — We don't have time to write a schema and validation checks; we need to ship our model now.

Without a contract, a schema change later will silently break the model's input, leading to bad predictions for weeks. Diagnosing that is far slower than the initial effort of writing a contract. Contracts reduce the most expensive part of data engineering: debugging integration issues. In fact: Data contracts are a small up-front investment that pays off in fewer outages, faster on-boarding of new consumers, and smoother schema evolution. They're a speed tool, not a burden.

**The semantic layer replaces the need for data contracts.** — We have a semantic layer that defines what 'revenue' means, so we don't need separate contracts for each table.

The semantic layer sits on top of the data. It assumes the data underneath is correct and clean. If the table that feeds the semantic layer changes its columns, the semantic layer breaks. A contract at the table level ensures the data is always in the expected shape, so the semantic layer can do its job. In fact: Data contracts are the foundation; the semantic layer is built on top of them. They complement each other: contracts guarantee the raw data's structure and quality, and the semantic layer guarantees the meaning.


## In short

A data contract is a promise about data: its shape, its meaning, its quality, and how it changes. It turns a silent breakage into a loud, negotiable change. Whenever you consume data from another system, or provide data to one, you need a contract. It's the first line of defence against garbage-in, garbage-out — and the foundation on which all trust between systems is built.


## Where this is used

Data contracts are everywhere in AI systems, and they're not optional.

**Large Language Models (LLMs):** Training data for an LLM is enormous — terabytes of text. Each data source (a web crawl, a book corpus, a forum) is a producer. The training pipeline is a consumer. If one source sends text that isn't UTF-8 encoded, or has HTML tags instead of plain text, the entire training run can be corrupted or wasted. Data contracts on the training data specify that each document must be text, in a certain encoding, with a certain minimum length, and with a document ID. The pipeline rejects any document that violates the contract, preventing garbage from polluting the model. For fine-tuning, the contract is even stricter: each example must have an 'instruction' and a 'response', both strings, non-empty. Without that, the model learns to produce nonsense.

**Small language models on devices (like a phone):** A small model on your phone, e.g., for next-word prediction or keyboard suggestions, is trained offline and then deployed. The training data comes from logs of user text. A contract specifies that each sample has a 'timestamp', a 'text' string, and a 'language' code. If the producer (the app) sends a sample without a language code, the training pipeline can't filter by language, and the model might be trained on mixed languages, degrading accuracy. The developer writes the contract, and the validation code rejects invalid samples before training.

**Retrieval-Augmented Generation (RAG):** In RAG, a model searches a knowledge base (like a set of documents) and uses the results to answer a question. The knowledge base is built from data contracts. Each document must have a 'title', 'body', and 'embedding' (a list of numbers representing its meaning). The contract ensures the embedding is the right length and the body is not empty. If a document lacks a body, the search retrieves it, the model reads it, and answers with nonsense. A contract prevents this by rejecting empty documents at ingestion.

**Agentic systems (that plan and call tools):** An agent calls an external API to get weather data, or to book a flight. The API's response must follow a contract: a 'temperature' as a number, a 'condition' as a string from a fixed set. If the API producer changes the 'condition' values ('rain' to 'precipitate') without updating the contract, the agent's logic that checks if condition == 'rain' fails silently. The contract, with a change process, forces the producer to negotiate the change, so the agent is updated first. In practice, the agent has schema validation code (like JSON Schema) that validates every API response before using it.

**Everyday applied ML (recommendations, vision, fraud detection):** A recommendation model takes as input user features (age, location, past purchases). The feature store (the producer) must follow a contract: 'age' is an integer between 0 and 120, 'purchase_amount' is a non-negative number. If the producer starts sending negative amounts due to a bug, the model makes bad recommendations or errors. The developer writes a validation step in the feature pipeline that checks the schema and range, and quarantines invalid records. For fraud detection, a contract on transaction data ensures every transaction has a 'time', 'amount', and 'merchant ID', and that the time is not in the future — future timestamps are a classic data bug that breaks the model.

**What a developer actually writes:** They write a schema definition (often in JSON Schema or Avro) that lists fields, types, and constraints. They write a validation function that checks records against the schema. They set up a test that runs on every change to the schema, checking that all historical data still conforms. And they set up a notification process: if a producer sends invalid data, the developer gets an alert, not just a silent failure. A common package is Great Expectations or an equivalent — it lets you write expectations like "column age must be >= 0" and runs checks on your data. Under the hood, that package does exactly the loop we wrote above: define rules, loop over rows, report failures. What breaks when done badly: one producer changes a field name from 'customer_id' to 'cust_id' without updating the contract. The consumer's SQL query says 'customer_id', gets no results, and the model serves empty recommendations for a day before anyone notices. That's a silent outage. The contract is the only thing that makes that a loud, actionable error.


---

Read this on the web: https://basavapurushottam.com/learn-ai/data-contracts
