# Imbalanced Data

The general problem of skewed distributions in training data, across classes, slices or time periods.

**Leads to:** Class Imbalance

**Rests on nothing.** A place to start.

*Machine Learning · Feature Engineering*


## Why learn this

This is the gateway to every real-world ML project. Before you can build a spam filter, a fraud detector, or a medical triage system, you must handle the fact that the rare event is the one that matters. Master this concept and you unlock the ability to evaluate models honestly, fix skewed training data, and avoid the trap where a model looks great but does nothing useful. It directly prepares you for 'Class Imbalance', 'Feature Selection', and eventually 'MLOps' — no serious model is built without confronting this problem first.


## The intuition

Imagine you are a security guard at a stadium entrance. You see thousands of fans a day, all with tickets. But every once in a while, someone tries to sneak in with a fake ticket. If you focus only on the 'normal' fans, you become great at letting them through, but you completely miss the fakes. An AI model does the same thing. It learns from the data you give it. If 99% of your data is 'normal' and only 1% is 'fake', the model quickly figures out it can score a 99% accuracy just by always predicting 'normal'. It becomes lazy. It never learns the subtle signs of a fake ticket. This is imbalanced data. It happens everywhere: fraud is rare, diseases are rare, equipment failures are rare. The problem isn't the rarity itself; it's that the model's goal — to minimise error — is misaligned with your goal, which is to catch the rare event. The model minimises error by ignoring the minority, but you don't care about the 99% it got right; you care about the 1% it missed.


## An analogy

Imagine you are a talent scout for a cricket team, and you have to decide which players are good enough to move up. You watch hundreds of local matches. But only a handful of players are truly exceptional — say 5 out of 500. If you just watch the matches and take notes, your brain naturally remembers the good players (the majority, who are average) and the exceptional ones (the minority) stand out. But an AI model training on your match notes would be like a scout who only records whether a player scored a century. If only 1% of players score a century, the model learns that 'century' almost never happens. So it predicts that no one will score a century. It's technically right 99% of the time, but it's useless to you. You want the model to find the 1% who score centuries, even if that means it sometimes flags a few average players as exceptional. That is the core tension: you accept more false alarms to catch the rare true positive. This analogy starts to break down, though, because a human scout uses intuition and context, whereas a model has no intuition — it only has numbers. The model cannot 'want' to find the rare player; it only does what the loss function tells it to do. That's why you, the engineer, have to change the loss function or the data.


## Definition

Imbalanced data is the situation where the classes (or categories) in a training dataset are not represented equally, with one class appearing much more frequently than others, which biases the model to favour the majority class unless corrected.


## How it connects

This concept sits at the heart of Feature Engineering, because the choice of features and the balance of the data interact strongly. Your notes on 'Class Imbalance' are the direct follow-up: it's the concrete problem of one class outnumbering others, and this page is the foundation for that. This concept also connects to 'Data Quality' — an imbalanced dataset is often a sign of poor data collection, not just a mathematical quirk. And it connects to 'Feature Selection' — if you have imbalanced data, some features may be unhelpful for the minority class, so you need to choose features that help separate the rare cases, not just the majority.


## The mathematics

To understand 'imbalanced data', we must look at how the classes in our dataset are distributed. In a balanced dataset, the number of examples in each class is roughly equal. In an imbalanced dataset, one class (the majority) has far more examples than the other (the minority).

We measure this imbalance using simple counts and division.

First, let us define the counts:
- Let N_minority be the number of samples in the smaller class.
- Let N_majority be the number of samples in the larger class.
- Let N_total be the total number of samples, which is N_minority plus N_majority.

The Imbalance Ratio compares the size of the minority class to the majority class. It is calculated as:
Imbalance Ratio = N_minority divided by N_majority.

For example, if there are 5 dogs and 95 cats, the ratio is 5 divided by 95, which is approximately 0.053. This means for every one dog, there are about 19 cats.

Why does this matter? Because many machine learning models are designed to maximize 'accuracy'. Accuracy is the fraction of total predictions that are correct. It is calculated as:
Accuracy = (Number of Correct Predictions) divided by N_total.

In an imbalanced dataset, a model can achieve high accuracy simply by always predicting the majority class. If a model predicts 'cat' for every single example in our dog/cat dataset, it will be correct 95 times out of 100. The accuracy is 95/100 = 0.95 or 95%. However, the model has failed to identify any dogs at all. This is why accuracy is misleading for imbalanced data.

To evaluate performance properly, we look at how well the model identifies the minority class. We use two metrics:

1. Recall (also called Sensitivity): This measures the fraction of actual minority cases that were correctly identified.
Recall = (True Positives) divided by (True Positives + False Negatives).
- True Positives (TP) are the minority cases correctly predicted as minority.
- False Negatives (FN) are the minority cases incorrectly predicted as majority.

2. Precision: This measures the fraction of predicted minority cases that were actually minority.
Precision = (True Positives) divided by (True Positives + False Positives).
- False Positives (FP) are majority cases incorrectly predicted as minority.

3. F1 Score: This is a single number that combines Precision and Recall. It is the harmonic mean of the two. The formula is:
F1 = 2 multiplied by (Precision multiplied by Recall) divided by (Precision + Recall).

The F1 score gives a balanced view of performance on the minority class, penalizing models that either miss many cases (low recall) or make many false alarms (low precision).


## A worked example

Let us walk through a concrete example with real numbers.

Scenario: We have a dataset of 1,000 credit card transactions. 
- 990 transactions are legitimate (Class 0, the majority).
- 10 transactions are fraudulent (Class 1, the minority).

Step 1: Identify the counts.
N_majority = 990
N_minority = 10
N_total = 990 + 10 = 1,000

Step 2: Calculate the Imbalance Ratio.
Imbalance Ratio = N_minority / N_majority
Imbalance Ratio = 10 / 990
Imbalance Ratio ≈ 0.0101
This tells us the minority class is about 1/100th the size of the majority class.

Step 3: Evaluate a 'Naive' Model.
Imagine a model that ignores the data entirely and always predicts 'legitimate' (Class 0).
- For the 990 legitimate transactions, it is correct. These are True Negatives (not relevant for minority metrics but part of accuracy).
- For the 10 fraudulent transactions, it predicts 'legitimate'. These are False Negatives.

Step 4: Calculate Accuracy for this Naive Model.
Number of Correct Predictions = 990 (all legitimate ones were correctly identified as legitimate).
Accuracy = 990 / 1,000
Accuracy = 0.99 or 99%.
At first glance, 99% accuracy seems excellent. But the model caught zero fraud cases.

Step 5: Calculate Recall for the Fraud Class (Class 1).
We need to know how many frauds we caught.
True Positives (TP) = 0 (we predicted no frauds).
False Negatives (FN) = 10 (all actual frauds were missed).
Recall = TP / (TP + FN)
Recall = 0 / (0 + 10)
Recall = 0 / 10
Recall = 0.
We caught 0% of the fraud cases.

Step 6: Calculate Precision for the Fraud Class.
Since we predicted no frauds at all, there are no False Positives either.
False Positives (FP) = 0.
Precision = TP / (TP + FP)
Precision = 0 / (0 + 0).
This is undefined because we made no positive predictions. In practice, if a model never predicts the minority class, its precision is considered 0 or irrelevant because it has nothing to be precise about.

Step 7: Calculate F1 Score.
Since Recall is 0 and Precision is effectively 0 (or undefined), the F1 score is 0.
F1 = 2 * (0 * 0) / (0 + 0) = 0.

Conclusion:
The naive model has 99% accuracy but 0% recall and an F1 score of 0. This demonstrates that accuracy is a poor metric for imbalanced data because it hides the complete failure to detect the minority class.


## From scratch, in Python

```python
# Teaching the concept: imbalanced data

# We'll start with a simple dataset: 1000 transactions, 10 fraudulent.

# In Python, we can use a list to hold numbers. A list is a collection that keeps things in order.
# Here is a list of 10 zeros and 990 ones? No, let's use a list of labels.

# Let's create a list of labels: 0 for normal, 1 for fraud.
# We'll imagine we have already read the data.

from collections import Counter

# Say we have loaded the labels into a list called 'labels'.
# For this teaching, let's create it manually, but in real life you would read from a file.

transactions = [0] * 990 + [1] * 10

# This is a list. A list is a container that holds items in order.
# [0] * 990 means a list with 990 zeros. Adding two lists concatenates them.

print("Number of transactions:", len(transactions))
print("Count of each class:", Counter(transactions))

# Counter is a dictionary that counts each unique item.
# A dictionary is a collection that maps keys to values. Here key is the class label,
# and value is how many times it appears.

minority_count = Counter(transactions)[1]
majority_count = Counter(transactions)[0]

imbalance_ratio = minority_count / majority_count
print("Imbalance ratio:", imbalance_ratio)

# Now we want to see the problem with accuracy.

# We will simulate a silly model that always predicts majority (0).

def always_majority_predict(labels):
    """This function returns a prediction for each label, always 0."""
    predictions = []
    for _ in labels:
        predictions.append(0)
    return predictions

# Let's understand the function above. We are using a 'for' loop.
# A loop repeats a block of code. Here, for each item in 'labels', we add a 0 to the predictions list.

predictions = always_majority_predict(transactions)

# Now compute accuracy: how many times we were correct.

correct = 0
for actual, predicted in zip(transactions, predictions):
    if actual == predicted:
        correct += 1

accuracy = correct / len(transactions)
print("Accuracy of always predicting majority:", accuracy)

# But we want to know how many frauds we caught. That's recall.

# Recall = True Positives / (True Positives + False Negatives)

TP = 0
FN = 0
for actual, predicted in zip(transactions, predictions):
    if actual == 1:
        if predicted == 1:
            TP += 1
        else:
            FN += 1

recall = TP / (TP + FN) if (TP + FN) > 0 else 0
print("Recall for fraud:", recall)

# Output: Accuracy is 99%, Recall is 0%. This shows the problem.
```


## The same thing, with a framework

```python
In real work, you would use scikit-learn. The same calculation is a few lines:

from sklearn.metrics import accuracy_score, recall_score

# assume y_true and y_pred are lists
print(accuracy_score(y_true, y_pred))
print(recall_score(y_true, y_pred))

But underneath, it is doing exactly what we did: counting true positives, false negatives, etc.
```


## What the code does

We built two lists: transactions and predictions. We used a list comprehension-like construction but here we just used multiplication. The Counter object counts each unique value. The always_majority_predict function is a function that takes a list and returns a list of zeros. A function is a reusable block of code; it takes inputs and returns outputs. Inside, we use a for loop to go through each element in labels. The zip function pairs up elements from two lists; we iterate over them. We compute accuracy and recall manually. Notice that we did not use any library beyond collections. This teaches you that the concept is arithmetic, not magic.


## Common misconceptions

**Accuracy is a good measure for imbalanced data** — My model got 99% accuracy, so it's working great.

If the data is 99% one class and 1% the other, a model that always predicts the majority class gets 99% accuracy, but catches zero of the rare class. Accuracy ignores the distribution. In fact: You need to look at precision, recall, and F1-score, or the ROC-AUC. Those tell you how well the model performs on the minority class.

**Imbalanced data can be fixed by just adding more data** — If I collect more data, it will balance out.

The rare event might be genuinely rare, so you will never collect enough. Adding more data may still be 99%/1%. In fact: You need active techniques like oversampling the minority, undersampling the majority, or changing the loss function to penalise missing the minority.

**Imbalance is a problem that only occurs in classification** — I'm doing regression, so I don't have imbalanced data.

Imbalance can also occur in regression when certain output ranges are rare, or in multi-label settings. In fact: Any distribution can be skewed. You need to check the distribution of your target values, not just classes.

**Synthetic oversampling always helps** — I'll use SMOTE to create more minority samples, then the problem is solved.

SMOTE creates synthetic points by interpolating between existing ones. In high-dimensional spaces or with noisy data, it can create artifacts that don't represent real data. In fact: Synthetic oversampling can help, but you must validate it carefully. Sometimes simple resampling with replacement or class weights is safer.

**Imbalanced means the data is bad** — My dataset is imbalanced, so I need to find better data.

In many real-world problems, imbalance is a natural characteristic of the problem (e.g., fraud is rare). The data is not bad; the question is how to model it. In fact: The imbalance is a property of the world. You need to design your model, evaluation, and features to handle it.


## In short

Imbalanced data is when one class is much more common than others. This makes a model lazy: it learns to predict the majority, which gives high accuracy but terrible performance on the important minority class. You must measure the imbalance (ratio), use appropriate metrics (precision, recall, F1), and fix it via resampling, class weights, or threshold tuning. Never trust accuracy alone.


## Where this is used

This concept is used in almost every real-world ML system. In large language models, it appears as the problem of rare tokens or rare phrases. A language model trained on internet text will see the word 'the' billions of times but a word like 'mnemonic' only rarely. If the model only optimised for the majority tokens, it would never learn to produce rare but important words. So they use techniques like temperature scaling and rare token sampling to mitigate the imbalance. In small language models running on a phone or a laptop, the problem is the same but the data is smaller; you have less data to see the rare cases, so you need to be even more careful. In retrieval-augmented generation (RAG), the concept appears in the embedding space. If your document collection has thousands of common articles and only one on a niche topic, the similarity search may always return the common articles, so the model never learns to retrieve the rare one. This is why you often rebalance the index or use query rewrite. In agentic systems that plan and call tools, the agent learns which tool to use. If 99% of the time it uses the weather tool and 1% it needs the calculator, the agent will default to the weather tool, leading to incorrect actions. To avoid this, you need to rebalance the training data or carefully tune the reward. In everyday applied ML, every fraud detection, anomaly detection, and medical diagnosis model faces this. A developer writes code to resample the data (e.g., using imbalanced-learn's SMOTE) or to assign class weights. They also change the evaluation metric from accuracy to F1-score or AUC. What breaks when done badly? The model reports a high accuracy, gets deployed, and then silently fails to catch any rare event, which can be catastrophic. For example, a hospital triage model that never flags a heart attack is 99% accurate but completely useless. A developer's day-to-day work involves looking at the distribution of labels, computing the imbalance ratio, and deciding whether to oversample, undersample, or use class weighted losses.


---

Read this on the web: https://basavapurushottam.com/learn-ai/imbalanced-data
