# Data Cleaning

Correcting or removing inaccurate, malformed and inconsistent records so downstream analysis rests on sound data.

**Rests on:** Handling Missing Data, Outlier Handling

**Leads to:** Data Literacy, Data Poisoning, Data Transformation, DNA Sequencing Analysis, Earnings Call Analysis, Null and Duplicate Handling, Unstructured Data

*Data Engineering and Analytics · Data Quality*


## Why learn this

Data cleaning is the foundation of every AI system. Before any model sees your data, the data must be trustworthy. This page gives you the skills to inspect, correct, and automate the fixing of your data. Once you master this, you'll be ready for the next steps in your library: **Data Transformation** (reshaping data for analysis), **Handling Missing Data** (deciding what to do with gaps), **Outlier Handling** (spotting extreme values), and eventually **Data Validation Constraints** (turning your cleaning rules into automated tests). In every project you'll ever build — whether a recommendation system, a spam filter, or a language model — data cleaning will be the majority of the work. This page is where you build the muscle.


## The intuition

Imagine you are cooking a meal for guests. You have a basket of vegetables, but some are bruised, some are stale, and some are not even vegetables — a few are small stones. You cannot cook everything as-is, or the dish will taste terrible. So you sort through the basket: you throw away the stones, you trim the bruises, you wash the dirt off the rest. Only then can you start cooking. Data cleaning is exactly that sorting and trimming, but for tables of numbers and text. Real-world data is never perfect. It arrives with typos, missing entries, inconsistent formats (one column says '1,234' and another says '1234'), and outright errors. If you skip cleaning and build a model on messy data, the model will learn the mess — it will make wrong predictions that look confident. Cleaning is not a chore that happens before the 'real' work; it *is* the real work, in terms of time and impact. This is why every serious data project spends the majority of its time here.


## An analogy

Think of a library. When a new shipment of books arrives, the librarian doesn't just put them on the shelves. She checks each one: is the title spelled correctly on the spine? Are the pages complete? Does the ISBN match the database? Is the book actually in the right category? If a book has a torn cover, she repairs it; if it's a duplicate, she may return it; if it's badly water-damaged, she discards it. She records the condition of each book before adding it to the catalogue. Otherwise, a reader searching for 'Python for Beginners' might find a book titled 'Pythn for Beginers' and get frustrated. The library's value depends on the catalogue being accurate. Now, the analogy breaks down in one important way: a librarian can personally inspect each book, holding it and flipping through pages. Data cleaning works with millions of rows — you cannot inspect each one by hand. Instead, you write rules and scripts that perform the inspection automatically. You define what 'good' looks like (e.g., age is a number between 0 and 120) and then use code to find rows that break the rule. The librarian's careful eye becomes your Python script. That’s the core skill: turning your judgement into automated checks.


## Definition

Data cleaning is the process of detecting and correcting (or removing) inaccurate, malformed, or inconsistent records in a dataset, so that the downstream analysis or model rests on sound, trustworthy data.


## How it connects

This concept is the operational heart of **Data Quality** — the parent concept that asks whether data is accurate, complete, timely, and consistent. Cleaning is how you *enforce* data quality. It also connects directly to your notes on **Data Dictionaries** (you need documented field definitions to know what 'correct' looks like), **Handling Missing Data** (cleaning often decides to drop or fill gaps), **Outlier Handling** (extreme values may be errors to remove or signals to keep), and **Data Transformation** (after cleaning, you often reshape the data). Cleaning is the first step after you have gathered raw data, and it feeds everything else.


## The mathematics

This concept draws on the simplest branch of mathematics: arithmetic. There are no derivatives, no algebra, no plots — just counting, comparing, and simple operations like adding, subtracting, finding averages, and checking if a number lies between two bounds. The 'mathematics' of data cleaning is really about patterns of missingness and distributions. For example, you might want to know: 'What percentage of rows have a blank in the age column?' That is a division: (number of missing) ÷ (total rows). Or you might ask: 'What is the average value in this column?' That is a sum divided by a count. You might also use a rule like the IQR (interquartile range) to flag outliers — that involves sorting numbers and finding the median. All of this is arithmetic, just organised in a careful way. The key mental operation is *comparison*: you compare each value to a standard (e.g., 'age must be between 0 and 120') and decide whether it passes. You are not inventing equations; you are applying simple checks and counting results.

Let the total number of rows in the dataset be N. Let M be the number of rows that contain at least one missing value. \[ \text{Missing percentage} = \frac{M}{N} \times 100\% \] Here N and M are plain counts—arithmetic only. A second formula: to check whether a numeric value x is within a plausible range [L, U], we test: \[ L \le x \le U \] If false, the value is flagged as an error. Every symbol (N, M, x, L, U) is just a name for a number you already know.


## A worked example

```
You have a small dataset of a library's patrons. The columns are: Patron ID, Age, and Books Borrowed. There are 5 rows. Some cells are missing, and one row has an impossible value.

Step 1: Row 1: ID=101, Age=25, Books=3   # Good row, nothing to fix.
Step 2: Row 2: ID=102, Age=, Books=1   # Age is missing. We count this as missing.
Step 3: Row 3: ID=103, Age=300, Books=0   # Age=300 is impossible. We flag this as an error, not missing.
Step 4: Row 4: ID=104, Age=30, Books=,    # Books is missing. Count it.
Step 5: Row 5: ID=105, Age=27, Books=5   # Good row.
Step 6: Missing count: 2 out of 5 rows = 2/5 = 0.4 = 40%   # Percentage of rows with at least one missing value.

Result: We have 3 valid rows, 2 rows with missing values, and 1 impossible value (age 300) that we must decide how to fix. The missing percentage is 40% — a high number that tells us the data needs serious attention before analysis.
```


## From scratch, in Python

```python
# Plain Python: reading a CSV and checking for missing values and impossibilities

# Let's create a small dataset manually, as a list of lists.
# Each inner list is one row: [id, age, books_borrowed]
# We use None to mean 'missing'.

data = [
    [101, 25, 3],
    [102, None, 1],
    [103, 300, 0],
    [104, 30, None],
    [105, 27, 5]
]

# We want to count rows that have at least one missing value (None)
# and also rows where age is impossible (outside 0-120).

missing_count = 0
error_count = 0
valid_rows = []

for row in data:
    age = row[1]
    books = row[2]
    
    row_has_missing = (age is None) or (books is None)
    row_has_error = (age is not None) and (age < 0 or age > 120)
    
    if row_has_missing:
        missing_count += 1
    if row_has_error:
        error_count += 1
    if not row_has_missing and not row_has_error:
        valid_rows.append(row)

print('Rows with missing values:', missing_count)
print('Rows with impossible age:', error_count)
print('Valid rows:', len(valid_rows))

# Now let's compute the percentage of rows with missing values
# Careful: 2/5 in Python 3 gives 0.4, not 0. So we multiply by 100.
total_rows = len(data)
missing_percent = (missing_count / total_rows) * 100
print('Missing percentage:', missing_percent, '%')

# Output:
# Rows with missing values: 2
# Rows with impossible age: 1
# Valid rows: 2
# Missing percentage: 40.0 %
```


## What the code does

Let's read the code line by line, because it introduces several Python basics. First, we create a list called `data`. A list is an ordered collection of items, written with square brackets `[...]`. Here, each item is itself a list of three numbers (or `None` to mean 'missing'). So `data` is a list of lists — like a small table. The first row is `[101, 25, 3]`, the second is `[102, None, 1]`, and so on. `None` is a special Python value meaning 'nothing' or 'missing'.

Next, we set up three counters: `missing_count`, `error_count`, and an empty list `valid_rows` that will hold the rows we keep. The empty list is written `[]`.

Then we use a `for` loop. A `for` loop repeats a block of code for each item in a collection. Here, `for row in data:` means 'take each row (one inner list) one by one, and run the indented block below with that row. We use `row[1]` to get the second element of the row (index 1 is the second position, because counting starts at 0). So `row[0]` is the ID, `row[1]` is the age, `row[2]` is the books.

Inside the loop, we check for missing values. The expression `(age is None) or (books is None)` uses `or` — it is true if either side is true. We store that result in the variable `row_has_missing`. Similarly, we check if age is an impossible number: `(age is not None) and (age < 0 or age > 120)`. This says: if age is not missing, and it is either less than 0 or greater than 120, then it's an error.

Then we have three `if` statements that act on these checks. The first: if the row has missing, add 1 to `missing_count`. The second: if the row has an error, add 1 to `error_count`. The third: if the row has neither, append the row to the `valid_rows` list using `.append(row)`.

Finally, we print the counts and compute the missing percentage. Division uses `/`. The print statement shows text and numbers.

This is the heart of data cleaning: writing simple checks (if statements) and counting results. In real projects, you would read data from a file instead of typing it, but the logic is identical.


## Common misconceptions

**Dropping rows with missing values is always the safe option** — If I just remove all rows that have any missing value, I won't have to think about it anymore.

This is a wrong belief because dropping is not always safe, as explained above. In fact: Missingness is a signal. Before dropping, you should explore *why* the data is missing. Sometimes you can fill in a sensible value (imputation), and sometimes you should keep the row and add a flag like 'income_missing = true' so the model can learn from the pattern.

**An outlier is always a mistake** — If a value is far from the average, it must be a data entry error, so I should remove it.

Outliers can be the most valuable observations. In fraud detection, a huge transaction might be the fraud you want to catch. In medical data, an extreme test result might indicate a real disease. Removing them blindly strips out the signal you care about. In fact: Examine outliers before removing. Check if they are plausible (e.g., a person age 300 is impossible, but age 90 is rare but possible). Use domain knowledge or a documented rule to decide. The IQR rule (values beyond 1.5× the interquartile range) is only a starting point, not a verdict.

**Cleaning is a one-time step done before the model** — Once I clean the data at the start, I can forget about it and focus on the model.

Real data changes. New rows come in with new errors. If your cleaning is not automated, the next time you load the data you'll have to do it all again, and you might do it differently (introducing inconsistencies). Also, the model may need the same cleaning at prediction time. In fact: Think of cleaning as a pipeline: a repeatable set of steps that runs every time new data arrives. Automate it. The same cleaning code should be applied to training data and to future data the model sees. This is why the notes say 'Automate it in the pipeline.' A cleaning step that is a manual script is fragile.

**An empty cell and a zero are the same thing** — If a cell is blank, I can just replace it with 0.

A blank likely means 'we never recorded this', while 0 often means 'the value really was zero' — e.g., a customer with zero purchases is different from a customer whose purchase count was never logged. Treating them the same distorts counts, averages, and model predictions. In fact: Distinguish 'missing' from 'zero'. In your data dictionary, define what an empty cell means. When cleaning, you might keep the missing as `NaN` (Not a Number) or fill it with a meaningful placeholder, but never blindly turn it into 0 unless you have a reason.


## In short

Data cleaning is the art of making data trustworthy. You check every value against what you know is correct, fix or remove what is wrong, and separate missing from zero. It is the biggest slice of any data project's time — and it is not a chore, it is the foundation. Automate your cleaning, and document every decision you make, because a cleaning rule is actually a modelling assumption. If you remember nothing else: clean early, clean often, and let the code do the boring work.


## Where this is used

Data cleaning is everywhere in AI. No model is trained without it. Here's how it shows up across the field.

**In large language models (LLMs):** The training data for an LLM — trillions of words scraped from the web — is a disaster of duplicates, boilerplate, spam, and missing context. Before training, the data is cleaned at scale: near-duplicate text is removed (because it would make the model repeat itself), boilerplate like 'click here' is stripped out, and sometimes whole web pages with too little content are dropped. Even after this initial cleaning, the model can still pick up biases or harmful text, so further cleaning filters by content. The cleaning logic is exactly the same as our little loop: check each document for patterns and decide to keep or discard. A developer working on an LLM pipeline uses tools like `pandas` (a Python library for tables) to load a CSV of URLs and document text, then writes filters that drop rows where the text length is below a threshold or where the URL is on a blocklist.

**In small models running on devices (e.g., on a phone or a Raspberry Pi):** When a model has to run in a tiny amount of memory, every byte matters. The data used to train it must be extremely clean, because the model has little capacity to 'absorb' noise. For example, a keyword-spotting model that listens for 'Hey device' on a smart speaker is trained on thousands of audio clips. If some clips contain background noise or are mislabeled (e.g., a clip of 'Hey device' labelled as 'noise'), the model learns to be confused. Cleaning here means removing misaligned audio clips and standardising the silence before the word. A developer would load an index of audio files, check that each one has the correct length, and remove ones that are too short or too long.

**In retrieval-augmented generation (RAG):** RAG lets an LLM answer questions by looking up relevant documents from a database. The quality of the answers depends entirely on the quality of that database. Before documents are embedded (turned into vectors), they are cleaned: duplicated paragraphs are dropped, broken headers are fixed, and empty chunks are removed. If a document has a garbled table of numbers, the embedding will be garbage, and the retrieval will pull up nonsense. Cleaning directly improves the retrieval step — a developer uses a library like `pandas` to load a list of document chunks, removes rows where the text is empty, and maybe uses a regular expression to strip HTML tags before embedding.

**In agentic systems (AI agents that plan and call tools):** An agent that can browse the web, query a database, or take actions needs trustworthy data at every step. If the agent reads a dataset of user transactions and one row says a purchase amount is negative (an error), the agent might make a decision based on that error — like approving a refund that shouldn't exist. Cleaning is built into the agent's data pipeline: before the agent's logic runs, the data is validated against constraints (e.g., 'amount must be greater than zero'), and rows that violate the constraints are flagged or removed. The agent's code calls a cleaning function, much like our loop, before it makes any decisions.

**In everyday applied machine learning (recommendation, vision, forecasting, fraud detection):** These are the most common cases. A recommendation system (like the one on a shopping site) is trained on user click histories. If a user session is recorded with a duplicated click event, the model may over-emphasise that item. Cleaning removes duplicate records. A fraud detection model sees millions of transactions; if a transaction has a missing merchant category, the model must either drop it or impute the category—this is a cleaning decision. A forecasting model for sales needs a clean time series: missing dates must be handled (either interpolated or marked), and outliers that are clearly data entry errors (an order of 100,000 units for a single customer) must be removed or corrected. In all these cases, the developer writes the same kind of code: load data, check each column for missing or impossible values, and apply a rule.


---

Read this on the web: https://basavapurushottam.com/learn-ai/data-cleaning
