# Data Dictionaries

Documentation defining each field's meaning, type, units, valid values and provenance.

**Rests on:** Data Literacy

**Leads to:** Data Validation Constraints, Regular Expression Constraints

*Data Engineering and Analytics · Data Quality*


## Why learn this

Before you can build any AI system — a chatbot, a recommendation engine, a model that spots fraud — you need data, and before you can use data you need to know what it means. Without a data dictionary, two people on the same team will interpret the same column in opposite ways: one thinks `salary` is in rupees per month, the other in lakhs per year. This concept gives you the tool to stop that confusion before it starts. You will use it in every project from your first exploratory analysis to the data pipelines that feed large language models. It is the foundation for the later topics in your library: Data Cleaning, Data Transformation, and Data Validation Constraints. Master this and you have a permanent answer to the question 'what does this field actually mean?'


## The intuition

Think of a data dictionary as an instruction manual for your data. A dataset is a table — rows are individual records (a customer, a transaction, a tweet) and columns are different pieces of information (name, amount, date). But the numbers and words in those columns mean nothing on their own. '42' could be an age, a score, or an ID. 'High' could be a category or a risk level. A data dictionary is the written explanation: it tells you, for each column, what the values represent, what type they are (number, text, true/false), what units they are in (meters, rupees, days), what values are allowed (only 0 or 1, or any number above 0), and where the data came from (from a form, from a sensor, from another database). Without it, you are guessing. With it, you can trust your analysis.


## An analogy

A data dictionary is like the legend on a map or the key to a crossword puzzle. Imagine you receive a map with squiggly lines, dots, and crosses, but no legend. You have no idea whether a dot means a city or a volcano, or whether the wavy line is a river or a road. The map is useless. The legend turns the marks into meaning. A data dictionary does the same for your dataset. But the analogy stops working in one important way: a legend is usually fixed and stays with the map. A data dictionary lives separately from the data, and it must be updated as the data changes. A map's legend rarely changes, but a column called 'status' might gain a new value ('pending') tomorrow, and if the dictionary is not updated, it is already out of date. So a data dictionary is not a one-time legend — it is a living document that must be maintained alongside the data it describes.


## Definition

A data dictionary is a written document that defines each field (column) in a dataset, specifying its meaning, data type, units, allowed values, and provenance (where the data came from), so that anyone using the data interprets it the same way.


## How it connects

This concept sits at the heart of Data Quality, which is the parent idea that data must be accurate, complete, timely, and consistent. Without a data dictionary, you cannot judge whether data is 'accurate' or 'consistent' because you have no agreed definition of what the fields mean. It directly supports Data Cleaning — when you correct malformed records, you are working against the dictionary's definition of what is valid. It also connects to Data Validation Constraints, which turn dictionary rules into automated tests. A data dictionary is the human-readable source; constraints are the machine-executable version. Finally, it is essential for handling Found Data — data that was collected for another purpose — because such data almost never comes with its own dictionary, so you must create one to understand its hidden biases.


## The mathematics

This concept is primarily about documentation and communication, so it does not rely on advanced mathematics. The only 'math' involved is basic arithmetic — counting and comparing values. The mathematical idea that underpins data quality is the notion of a 'type': whether a value is a number, a piece of text, or a true/false flag. You do not need to calculate anything, but you need to be precise about what kind of value each field holds. For example, the field 'age' is a number (integer), 'name' is text (string), 'subscribed' is a boolean (true/false). The data dictionary states these types explicitly, and when you later write code to validate data, you will compare each value against the declared type — for instance, checking that 'age' is never a word like 'old'. That comparison is a form of arithmetic: checking whether a value belongs to a set. This concept teaches you to think in terms of sets and categories, which is the groundwork for how computers handle data.

Type: T ∈ {number, text, boolean, date, ...}  Units: U (e.g., ₹, cm, kg)  Allowed values: V = {v1, v2, ...}  Provenance: P (source description). A well-formed data dictionary entry for a field f is a tuple: (meaning, type, units, allowed_values, provenance).


## A worked example

```
You have a small table with three columns: Customer ID, Amount (in ₹), and Location. Here is the data for three customers: (C1, 5000, Mumbai), (C2, 12000, Delhi), (C3, 3000, Mumbai). Without a dictionary, what does 'Amount' mean? Is it in rupees or paise? Is it monthly or yearly? Is it before or after tax? The dictionary resolves this.

Step 1: Define each field's type: Customer ID → text (string) or number?   # Customer ID is usually an identifier, not a number you would add or multiply. So it is text, even if it looks like digits. The dictionary says 'string'.
Step 2: Amount → number (integer), unit = Indian Rupees (₹)   # The dictionary specifies the type (integer) and the unit (₹). This prevents someone from thinking it is in lakhs or dollars.
Step 3: Location → text (string), allowed values = list of Indian cities   # The dictionary can restrict allowed values, e.g. only 'Mumbai', 'Delhi', 'Bengaluru', etc. Any other entry is invalid.

Result: With the dictionary, anyone reading the data knows that C1's amount is 5,000 rupees, not 5,000 paise, and that 'Mumbai' is a valid city, not a typo. The dictionary resolves ambiguity.
```


## From scratch, in Python

```python
# No special libraries needed. Plain Python.

# A data dictionary can be represented as a list of dictionaries.
# First, what is a dictionary in Python? It is like a real-life dictionary:
# you look up a word (a 'key') and get its definition (a 'value').
# For example:
# my_dict = {'name': 'Alice', 'age': 30}
# Then my_dict['name'] gives you 'Alice'.

# Here we define a data dictionary for a small dataset.
# Each entry is a Python dictionary with the fields we defined.

data_dictionary = [
    {
        "field": "customer_id",
        "meaning": "Unique identifier for each customer",
        "type": "string",  # text, not a number you can do arithmetic on
        "units": None,   # no units for an ID
        "allowed_values": None,  # any string, but must be unique
        "provenance": "Generated by the order system",
    },
    {
        "field": "amount",
        "meaning": "Total purchase amount",
        "type": "number",  # integer or float
        "units": "INR",  # Indian Rupees
        "allowed_values": ">= 0",  # can't be negative
        "provenance": "From the payment gateway",
    },
    {
        "field": "location",
        "meaning": "City where the customer is based",
        "type": "string",
        "units": None,
        "allowed_values": ["Mumbai", "Delhi", "Bengaluru"],  # only these are valid
        "provenance": "From the registration form",
    },
]

# Let's print it out nicely.
for entry in data_dictionary:
    print(f"Field: {entry['field']}")
    print(f"  Meaning: {entry['meaning']}")
    print(f"  Type: {entry['type']}")
    print(f"  Units: {entry['units']}")
    print(f"  Allowed values: {entry['allowed_values']}")
    print(f"  Provenance: {entry['provenance']}\n")
```


## The same thing, with a framework

```python
In practice, especially with libraries like pandas (which you may meet later), you might store a data dictionary as a DataFrame or as a separate JSON file. But the underlying idea is the same. The 'from scratch' code shows the plain Python way; a library like pandas would just let you hold the dictionary in a table, but it is still a list of fields with their attributes.
```


## What the code does

Line by line:  First, we create an empty list called `data_dictionary` to hold our entries. Then we add three dictionaries to it. Each dictionary has keys: 'field', 'meaning', 'type', 'units', 'allowed_values', 'provenance'. We use `None` for units and allowed values when they don't apply (like customer_id has no units). For allowed_values, we can put a list of valid strings, e.g., ['Mumbai', 'Delhi', 'Bengaluru']. Finally, we loop over the list with `for entry in data_dictionary:` and print each field's details. The `f"..."` strings let us insert the values directly. This is a minimal example, but in a real project you would write this dictionary to a JSON or YAML file so it can be read by other tools.


## Common misconceptions

**A data dictionary is only for databases** — Data dictionaries are only needed in large corporate databases, not for my small project or dataset.

Even a tiny dataset with two columns has enough ambiguity to cause errors. A three-row table with a 'date' column becomes unusable if you don't know whether it is day-first or month-first. The size of the data does not determine the need for a dictionary. In fact: Every dataset, no matter how small, benefits from a dictionary. It is the cheapest way to prevent misinterpretation. A two-column CSV from a friend still needs a dictionary.

**A data dictionary is optional if you wrote the data yourself** — Because I created the data, I know what every field means, so I don't need to write it down.

Memory fades. Six months later, you will not remember whether 'amount' includes tax or not. And if you ever share the data with a colleague, the dictionary is the only way they can know. In fact: Write the dictionary at the moment of creation. It takes little time and saves hours of confusion later. It is a form of self-documentation that everyone, including future you, will thank you for.

**The data dictionary is the same as the database schema** — I already have a schema in my database, so I don't need a separate data dictionary.

A schema defines data types and constraints (e.g., column 'age' is an integer) but it does not explain meaning, units, or provenance. You can have an integer column and still not know whether it is in years or months. In fact: A schema is a subset of a data dictionary. A full dictionary adds the human-level meaning: what the field represents, its units, valid business values, and where the data came from.

**Once written, a dictionary never changes** — I write the dictionary once when I create the dataset, and I never have to update it.

Data evolves. New categories appear (a new city in the 'location' field), units may change (you switch from rupees to dollars), or the meaning of a field gets refined. A static dictionary quickly becomes outdated. In fact: A dictionary is a living document. It must be updated whenever the data definition changes. Best practice is to version control it, just like your code.


## In short

A data dictionary is the written explanation of every column in your dataset: what it means, what type it is, its units, what values are allowed, and where it came from. It turns a table of ambiguous numbers and strings into trustworthy data. Use it at the start of every project, update it when the data changes, and share it with everyone who works with the data. It is the foundation of data quality and the first line of defence against silent, costly errors.


## Where this is used

Data dictionaries are not just a nice-to-have; they are the backbone of every data-driven AI system. In large language models (LLMs) like ChatGPT, the training data is huge and unstructured text, but when you fine-tune a model or build a retrieval-augmented generation (RAG) system, you feed it structured data that needs a dictionary. For example, if you build a RAG system to answer questions about your company's products, the product database has fields like 'price', 'stock', 'description'. Without a dictionary, the model might think 'price' is in dollars when it is in rupees, or that 'stock' means inventory count when it could mean stock price. A developer would write the dictionary to a JSON file and then load it in code to validate incoming data before it enters the system. In small models running on edge devices (like a keyword-spotter on a phone), the dictionary defines the input features (e.g., audio sample rate, number of frames) and the output classes (e.g., 'yes', 'no'). If the dictionary is wrong, the model crashes or produces garbage. In agentic systems that plan and call tools, the agent needs to know the schema of the data it reads — a data dictionary provides that. For instance, an agent that schedules meetings needs to know that a 'date' field is in ISO format, not 'DD/MM/YYYY'. In everyday applied machine learning like recommendation or fraud detection, the dictionary is the first thing a data scientist writes. When a feature like 'transaction_amount' arrives, they check it against the dictionary: it must be a positive number, in INR, no missing values. Without it, the model silently sees corrupted data and learns the wrong patterns. What breaks when a dictionary is missing? A classic disaster is when one team labels a column 'salary' and another labels it 'compensation', and they merge datasets without realising they are the same concept but with different units — one in lakhs per year, the other in thousands per month. The model trains on mismatched scales and produces nonsense. A dictionary prevents this by forcing everyone to agree on names, units, and meanings before the data ever reaches a model.


---

Read this on the web: https://basavapurushottam.com/learn-ai/data-dictionaries
