# Handling Missing Data

Deciding what to do about absent values — drop, impute, or model the missingness itself.

**Leads to:** Data Cleaning, Null and Duplicate Handling

**Rests on nothing.** A place to start.

*Data Engineering and Analytics · Data Quality*


## Why learn this

You have a spreadsheet of customer records. Three customers have no listed income. One has no age. Before you can calculate an average income or plot trends, you must decide what to do with these blanks. This decision unlocks the rest of your work—cleaning, transformation, and modeling—because every other step assumes you have handled absence first. Get this wrong, and every number you produce is quietly wrong. In a recent audit, a team ignored missing income values and overestimated their average customer spend by 15%, leading to flawed budget forecasts. In another case, deleting records with missing age skewed the results toward older demographics. This is your first genuine modeling decision: what happens when data is absent?


## The intuition

Think of a spreadsheet where some cells are empty. These aren't just blanks; they are information. The income column might be empty for people who are self-employed because they did not want to share that number. The age column might be empty for people who signed up via an app that never asked for their birthday. The pattern of emptiness reveals how the data was collected, and that pattern matters as much as the values themselves. Before you fill in a gap, ask: why is this gap here? Is it random, like a survey respondent accidentally skipping a question? Or is it systematic, like a sensor that only fails when the temperature is extreme? If income is missing randomly, you might safely fill it with a calculated guess (the average of other incomes). If age is missing because the app never asked, deleting those records entirely might be safe. But if the absence itself is significant—say, people hide their salary when they earn too much—you might treat that absence as a signal rather than a gap to fill.


## An analogy

Imagine you are a chef with a recipe calling for three ingredients: flour, sugar, and eggs. You open the pantry and find you are out of sugar. You have three options. First, you could skip the sugar entirely and proceed—this is dropping the affected records. Second, you could substitute honey that you have on hand—this is replacing the blank with a calculated guess (imputation). Third, you could change the recipe to a sugar-free version altogether, acknowledging that the missingness changes what you are making—this is modeling the missingness. The analogy works because the right choice depends on context. If you are making bread, skipping sugar changes the texture but the bread will still rise. If you are making meringue, skipping sugar means the egg whites will not whip. Where the analogy breaks: in cooking, you can usually taste as you go and adjust. In data, you cannot 'taste' the dataset; you only see the final result after the analysis is done, and by then it is too late to fix a bad choice.


## Definition

Handling missing data means deciding whether to delete records with gaps, replace empty cells with calculated guesses, or treat the absence of data as a meaningful signal in itself. The choice depends on why the data is missing and what you are trying to measure.


## How it connects

This concept sits inside Data Quality, which explores how missing data patterns carry their own signal. Your notes mention Data Cleaning and Data Transformation; mastering missing data ensures you handle it before scaling or encoding values, avoiding the common error where spreading a problem distorts results. This also connects to Data Validation Constraints, which are the rules that prevent such gaps from appearing in future records.


## The mathematics

This concept relies on basic arithmetic: adding numbers, counting them, and dividing. The core idea is that when you guess a missing value (a process called imputation), you are making a specific mathematical statement about what the data looks like. You must choose your method carefully because each choice changes the properties of your dataset.

**Method 1: Mean Imputation**
The simplest way to handle a blank cell is to fill it with the average (mean) of the values that are actually present in that column. The mean is calculated by adding up all the known numbers and dividing by how many numbers there are.

Let us define the known numbers in a column as x_1, x_2, ..., x_n. Here, n is simply the count of non-missing entries. The symbol x-bar (read 'x-bar') represents the mean.

The formula for the mean is:
x-bar = (x_1 + x_2 + ... + x_n) divided by n

When you replace every blank in that column with x-bar, you are asserting that the missing values are exactly like the average of the observed ones. The mathematical consequence is that the overall average of the full dataset remains unchanged. However, this method reduces the spread (variance) of your data because you have replaced diverse real numbers with identical copies of the average. This can make your results look more precise than they really are.

**Method 2: Grouped Imputation**
A slightly more accurate approach uses information from another column. For example, if you are guessing a missing income value, you might group the data by city first. You calculate the mean income for each city separately using only the people in that city who reported their income. Then, you fill the blank for a person in 'City A' with the average income of 'City A'. This uses more information and distorts the data less than using a single global average.

**Method 3: Missingness Indicator**
Instead of guessing the number, you can keep the original value blank (or set it to zero) and add a new column that records whether the value was missing. For each row i, define a new value m_i. If the original value is missing, m_i equals 1. If the original value is present, m_i equals 0. This treats the absence of data as a signal in itself. Mathematically, this does not change the distribution of the numeric values but adds a binary feature that may correlate with the outcome.

**Method 4: Dropping Data**
You can also choose to remove any row that has a blank value. This is mathematically simple: you just reduce your sample size n. The risk is that if the missingness is not random, the remaining data may no longer represent the whole population.


## A worked example

```text
You have a small dataset of 5 customers. Column 'age' has values: 25, 32, blank, 28, blank.
You want to handle the missing values before analysing average age.

Step 1: Identify present values.
The ages that are actually recorded are 25, 32, and 28. Ignore the blanks for now.

Step 2: Sum the present values.
Add the known numbers together:
25 + 32 + 28 = 85

Step 3: Count the present values.
There are 3 non-missing entries. So, n = 3.

Step 4: Calculate the mean.
Divide the sum by the count:
Mean (x-bar) = 85 divided by 3 = 28.333...
We will use 28.33 for simplicity.

Step 5: Option A - Mean Imputation.
Replace each blank with the calculated mean (28.33).
The new list of ages is: 25, 32, 28.33, 28, 28.33.
Check the new average:
Sum = 25 + 32 + 28.33 + 28 + 28.33 = 141.66
Count = 5
New Mean = 141.66 / 5 = 28.33.
The average stays the same, but all values are now closer to the center. The spread is smaller.

Step 6: Option B - Drop Rows.
Remove the rows with blanks entirely.
Remaining ages: 25, 32, 28.
Count = 3.
Mean = 85 / 3 = 28.33.
The average is still 28.33, but you have lost 40% of your data points (2 out of 5). If the missing ages were systematically different (e.g., older people refused to answer), this remaining group is no longer representative.

Step 7: Option C - Missingness Indicator.
Keep the original numbers (or set blanks to 0) and add a new column 'is_missing'.
Row 1: Age 25, is_missing = 0
Row 2: Age 32, is_missing = 0
Row 3: Age blank, is_missing = 1
Row 4: Age 28, is_missing = 0
Row 5: Age blank, is_missing = 1
This preserves the original numeric distribution and adds a binary signal that might help predict other outcomes.

Result Summary:
- Mean imputation keeps all rows but artificially reduces variability.
- Dropping data loses information but avoids inventing numbers.
- Missingness indicators preserve the numeric data while flagging the absence as a feature.
```


## From scratch, in Python

```python
import csv

def read_data(filename):
    with open(filename, 'r') as f:
        reader = csv.DictReader(f)
        rows = list(reader)
    return rows

def mean_of_present(values):
    present = [float(v) for v in values if v != '']
    if len(present) == 0:
        return None
    return sum(present) / len(present)

def impute_mean(rows, column):
    values = [row[column] for row in rows]
    mean = mean_of_present(values)
    if mean is None:
        return rows
    for row in rows:
        if row[column] == '':
            row[column] = mean
    return rows

def add_missingness_column(rows, column):
    new_column = column + '_missing'
    for row in rows:
        row[new_column] = '1' if row[column] == '' else '0'
    return rows

def main():
    rows = read_data('customers.csv')
    rows = add_missingness_column(rows, 'age')
    rows = impute_mean(rows, 'age')
    for row in rows:
        print(row)

if __name__ == '__main__':
    main()
```


## The same thing, with a framework

```python
import pandas as pd

df = pd.read_csv('customers.csv')

df['age_missing'] = df['age'].isna().astype(int)

mean_age = df['age'].mean()
df['age'].fillna(mean_age, inplace=True)

print(df)
```


## What the code does

Let's go through the code step by step. First, we import `csv`, which is a built-in Python module for reading and writing CSV files — Comma-Separated Values, the most common format for tabular data. The `read_data` function opens a file, creates a `csv.DictReader` that turns each row into a dictionary (a mapping from column names to values), and returns a list of those dictionaries. A list is an ordered collection, and a dictionary lets you look up a value by its key, like `row['age']`.\n\nThen we define `mean_of_present`, which takes a list of values (as strings) and filters out the empty strings using a list comprehension: `[float(v) for v in values if v != '']`. This is a compact way to loop over each value, skip the blanks, and convert the rest to numbers (float is a number with a decimal point). If there are no present values, we return `None` (Python's way of saying 'no value'). Otherwise, we compute the mean by summing and dividing by the count.\n\n`impute_mean` does the actual filling: it looks at the column, computes the mean of present values, and then loops over each row; if that row's value is empty, it sets it to the mean. `add_missingness_column` adds a new key to each dictionary, '_missing', with '1' if the original was empty and '0' otherwise.\n\nFinally, `main` ties it together: read the data, add the missingness column, impute the mean, and print each row. This shows both main strategies in action.


## Common misconceptions

**Mean imputation is a safe default** — I can just fill missing values with the average and the data will be fine.

Mean imputation reduces the variability in the data, making the distribution look tighter than it is. This can shrink standard errors, inflate t-statistics, and lead you to think a relationship is significant when it isn't. It also ignores the possibility that missing values are systematically different from present ones. In fact: Mean imputation is convenient but distorts variance and relationships. Prefer methods that preserve variability, like regression imputation or multiple imputation, or use the missingness indicator.

**Drop all rows with missing values** — The simplest and most honest approach is to delete any row with a missing cell.

Dropping rows can remove a large portion of your data, especially if missingness is widespread. If the missing data is not random, dropping the rows biases your sample — you're keeping only the records that happened to be complete, which may not represent the population. In fact: Dropping is acceptable only when the number of missing rows is small and the missingness is random. Otherwise, impute or model the missingness.

**Missing means zero** — If a value is missing, I can just treat it as zero.

Zero is a meaningful number. A missing income is not an income of zero — that's a completely different claim. Treating missing as zero introduces a systematic bias, especially if the true value is positive. In fact: Zero is a substantive value, not a placeholder for absence. Treat missingness as its own category, and only map to zero if there's a domain reason to do so.

**Missing data is always a mistake** — Missing values are always errors in data entry, so I should just clean them up.

Missingness is often a meaningful signal. For example, a customer who doesn't provide income may be self-employed or concerned about privacy. The very act of not answering can correlate with the outcome you're studying. In fact: Missingness is information. Always investigate the pattern of absence before deciding on a strategy.

**Any imputation is better than none** — I have to fill in every gap, otherwise I can't run the analysis.

Blindly imputing can introduce more bias than leaving the data alone. For example, using a global mean imputation when missing values cluster in a subgroup can swamp the actual patterns in that group. In fact: Imputation is a modelling assumption. Sometimes the best approach is to keep the missingness and use a model that can handle it, like a tree-based algorithm.


## In short

Missing data is unavoidable, but how you handle it is a decision that shapes every downstream result. The three main approaches are: drop the rows, impute a value, or model the missingness itself. The right choice depends on why the data is missing — random absence can be imputed, systematic absence might be a signal. Always assess the pattern before acting, and document your choice. Some models (like tree-based ones) can handle missing values natively; others require pre-processing. The key is to see missingness as information, not a problem to be erased.


## Where this is used

Handling missing data shows up in every corner of AI development, and often the stakes are higher than they seem.\n\n**Large language models (LLMs).** LLMs are trained on enormous text corpora where sentences and documents are often truncated or incomplete. Before tokenization, the pipeline must decide what to do with missing or malformed text — for example, a URL that was cut off mid-way. The standard practice is to drop the malformed samples entirely, because a half-sentence provides little signal and could mislead the model. A more subtle form of missingness is in structured metadata: if a dataset of code snippets lacks the 'programming language' field, the model may be trained without that crucial hint, and the absence itself becomes a pattern to learn. Developers using LLMs for downstream tasks also face missing data in their own inputs: if you're building a chatbot that answers from a knowledge base, missing entries in the base should be handled before retrieval, or the model will confidently hallucinate a reply.\n\n**Small models on devices (TinyML).** On a phone or a smartwatch, sensors occasionally fail — a battery-saving mode might switch off the GPS, or a heart-rate monitor might lose signal during exercise. These gaps are often short, and the model must decide in real time what to do. Imputation is rarely possible because you can't wait for a batch; instead, you use the last known value (forward-fill) or mark the missingness with an indicator. The missingness indicator is especially valuable here: a model that knows 'GPS was off' can adapt its behaviour, e.g., relying more on the accelerometer. The constraint is computation and storage — you can't run a complex imputation model on a device, so you rely on simple heuristics.\n\n**Retrieval-augmented generation (RAG).** RAG systems answer questions by first retrieving relevant documents from a database, then feeding them to an LLM. The retrieval step often relies on embeddings — vectors that represent meaning — and those embeddings are computed from text. If a document is missing its title or has an empty body, you have a data quality problem. The pragmatics: you either drop the document from the index (so it's never retrieved) or you index it with a placeholder embedding (e.g., a vector of zeros). The missingness indicator can also be a feature in hybrid retrieval systems that combine lexical and semantic search. Developers building RAG pipelines often write code to filter out documents with missing critical fields before embedding, because a bad document in the index can poison the answer.\n\n**Agentic systems that plan and call tools.** An AI agent that browses the web or calls APIs will frequently encounter missing fields in the responses — a tool might return a JSON object without a 'status' key, or a web page might not have a 'text' section. The agent must decide how to proceed: does it treat the missing field as an error and retry, or does it proceed with a default? Agents are essentially decision loops, and each missing value is a decision point. Poor handling can cause the agent to crash or, worse, to make a wrong decision based on a false assumption. A robust agent will have explicit logic for missing data — checking if a key exists before using it, and logging the absence as a failure mode. This is a form of modelling the missingness, because the agent's behaviour changes when it knows data is absent.\n\n**Everyday applied machine learning.** In tabular data — the bread and butter of most ML work — missing values are common. Credit scoring: a bank's dataset might have missing 'income' for some applicants. The choice of imputation directly affects whether the model approves a loan. Fraud detection: a missing 'transaction amount' is probably a data entry error, and dropping the row is safe; a missing 'device ID' might indicate a fraudster using a fresh device, so the missingness itself is a powerful signal. Medical diagnosis: a missing lab result is not random — it could mean the test wasn't ordered, which is informative. The developer's workflow is usually: (1) assess missingness with a heatmap or `df.isnull().sum()`; (2) decide per column, not globally; (3) implement a pipeline that handles missing values before feature scaling; (4) test that the pipeline isn't leaking information (e.g., computing the mean on the full dataset before splitting can leak the target). If done badly, you get silently incorrect predictions: mean imputation that hides variance, or dropping so many rows that the remaining data is no longer representative.\n\n**What a developer actually writes.** In Python, the most common tools are `pandas` for dataframes, `scikit-learn` for machine learning pipelines, and `numpy` for numerical operations. The critical piece is a `Pipeline` object that includes a transformer for missing values, such as `SimpleImputer(strategy='mean')` or `KNNImputer`. The mistake beginners make is to impute before splitting into train and test sets, which leaks information — the mean should be computed only on the training fold. In production, missing data is handled in the data ingestion layer, often with a schema that defines which fields are required. Tools like Great Expectations or Deequ allow you to enforce that no missing values pass into the model, blocking the pipeline if they do.\n\nIn summary, missing data is not a nuisance to be swept under the rug; it's a design decision. Choose your method based on the meaning of the missingness, and document it — because every imputation is a modelling assumption.


---

Read this on the web: https://basavapurushottam.com/learn-ai/handling-missing-data
