# Rare Event Prediction

Predicting outcomes that occur very infrequently, where extreme class imbalance breaks default assumptions across the board.

**Rests on:** Cross-Entropy Loss, Logistic Regression

**Leads to:** Churn Prediction

*Machine Learning · Machine Learning Foundations*


## Why learn this

You likely deal with situations where the thing you want to prevent or find happens very rarely, but getting it wrong is expensive or dangerous. Imagine your company spends money on advertising, and only 1 in every 1,000 people who see an ad actually buys something. If your system flags everyone as a buyer, it is 'correct' 99.9% of the time, but you waste all your ad budget on people who won't buy. If you ignore everyone, you lose all sales. The core challenge is that standard rules fail here because 'being right most of the time' is meaningless when the minority event is what matters. Understanding this helps you decide whether to prioritize catching rare defects (safety) or avoiding wasted resources (cost), turning a vague business problem into a precise calculation.


## The intuition

Picture yourself in charge of a factory that makes airplane engine parts. Every hour, thousands of parts come off the line. Almost all of them are fine — only about one in ten thousand is defective. Your job is to build a system that flags the defective parts before they are installed, because a faulty part could cause a crash. The defect is a rare event.

Now, imagine you build a simple system that just says 'all parts are fine.' It would be right 99.99% of the time. That sounds amazing on paper, but it is completely useless — you never catch the one bad part. This is the trap of rare events: the thing you care about happens so rarely that a model can be 'correct' almost always by doing nothing.

So, you cannot use standard accuracy to judge your model. You must look at two specific mistakes:
1. Missing a defect (calling a bad part 'fine'). This risks a crash.
2. Flagging a good part as defective (calling a fine part 'bad'). This wastes money on unnecessary inspections.

The art of rare event prediction is balancing these two costs. If crashes cost billions, you set your system to be very sensitive, accepting that you might waste time inspecting many good parts. If inspections are extremely costly and crashes are unlikely, you raise the bar for what counts as a defect. The 'threshold' — the cutoff point where a part changes from 'fine' to 'defective' — is not fixed; it is a decision based on which error hurts more.

This leads to a crucial practice: never trust your model's performance on the data you used to build it. You must split your data into three groups:
1. Training data: The examples you use to teach the system how to spot defects.
2. Validation data: A separate set of examples used to tune the sensitivity (threshold) without showing the system new information during learning.
3. Test data: A final, hidden set of examples used only once at the very end to see if the system works on truly unseen data. This process ensures your model hasn't just memorized the factory's specific parts but has learned the general features of a defect.


## An analogy

Think of a smoke detector in a house. A real fire is rare. The detector can be set to be very sensitive (goes off at the faintest smoke) or less sensitive (only big flames). If it is too sensitive, every time you cook something, the alarm goes off — that is a false positive. If it is not sensitive enough, a real fire might go undetected — that is a false negative.

The cost of a false negative is potentially catastrophic (the house burns down), so we usually set the detector to be quite sensitive, accepting some false positives (burnt toast) as the price. But in a different setting, like a server room, a false alarm might cause a whole building to be evacuated, so we would set it differently. The threshold is not a universal constant; it is a business decision that depends on the relative costs.
The analogy stops working when we consider that a smoke detector has a single threshold, but a machine learning model has many parameters, so we can also adjust how it weighs different features. But the fundamental idea of a threshold that balances errors is exactly the same.


## Definition

Rare event prediction is the task of forecasting outcomes that occur very infrequently, where the extreme imbalance between the rare class and the common class breaks the default use of accuracy as a metric and forces a careful choice of decision threshold based on the asymmetric costs of false positives and false negatives.


## How it connects

This concept sits at the foundation of Machine Learning Foundations, which you have not yet studied — but it introduces the key idea that generalisation (performing well on new, unseen data) is the goal, not training accuracy. It connects directly to your later topics: Churn Prediction (where only a small percentage of customers churn), Crop Disease Detection (where disease is rare in the field, just like our defects), and Stock Price Prediction (where a profitable move is a rare event). The concept also introduces the train-validate-test discipline you will use in every model, and the cost asymmetry that makes simple accuracy useless in many real-world problems. You will see this pattern again and again: whenever the event you care about is rare, the default assumptions about how to evaluate a model break down.


## The mathematics

When an event is very rare, standard accuracy is misleading. If you predict 'no event' for every case, your accuracy will be nearly 100%, but you have learned nothing about the rare event. To measure performance on the rare event, we count four specific outcomes from our test data.

First, we define the counts using plain words and simple addition.

True Positives (TP): The number of times the model said 'event happened' and it actually did happen.
False Positives (FP): The number of times the model said 'event happened' but it did not happen. This is a false alarm.
True Negatives (TN): The number of times the model said 'no event' and there was indeed no event.
False Negatives (FN): The number of times the model said 'no event' but the event actually did happen. This is a missed detection.

These four numbers must add up to the total number of test cases. If they do not, one of the counts is wrong.

From these four counts, we calculate three rates. Each rate is a fraction: a numerator divided by a denominator.

The True Positive Rate (also called Recall) tells us how many of the actual rare events we found. We calculate it by taking the number of True Positives and dividing it by the total number of actual events. The total number of actual events is the sum of True Positives and False Negatives. So, True Positive Rate equals TP divided by (TP plus FN).

The False Positive Rate tells us how often we falsely alarm when there is no event. We calculate it by taking the number of False Positives and dividing it by the total number of actual non-events. The total number of actual non-events is the sum of True Negatives and False Positives. So, False Positive Rate equals FP divided by (FP plus TN).

Precision tells us how reliable our alarms are. If we sound an alarm, what is the chance it is real? We calculate it by taking the number of True Positives and dividing it by the total number of times we sounded an alarm. The total number of alarms is the sum of True Positives and False Positives. So, Precision equals TP divided by (TP plus FP).

Accuracy is the fraction of all correct predictions. It is calculated as (TP plus TN) divided by the total number of cases (TP plus FP plus TN plus FN). As shown in the example below, accuracy can be high even when the model fails to find the rare event.

Symbol glossary:
- TP: number of true positives
- FP: number of false positives
- TN: number of true negatives
- FN: number of false negatives
- TPR: true positive rate
- FPR: false positive rate


## A worked example

Suppose a factory produces 10,000 parts. Only 10 of these parts are defective. The defect is the rare event. A model inspects every part and predicts whether it is 'defective' or 'okay'. Here are the results:

Step 1: Count the True Positives (TP). These are the defective parts that the model correctly identified as defective. In this case, TP = 7.

Step 2: Count the False Negatives (FN). These are the defective parts that the model missed (it said 'okay'). In this case, FN = 3. Note that TP plus FN equals 10, which is the total number of actual defective parts.

Step 3: Count the False Positives (FP). These are the good parts that the model incorrectly flagged as defective. In this case, FP = 50.

Step 4: Count the True Negatives (TN). These are the good parts that the model correctly identified as okay. The total number of good parts is 10,000 minus 10, which is 9,990. Since 50 were falsely flagged, the remaining good parts are TN = 9,940. Note that FP plus TN equals 9,990, which is the total number of actual non-defective parts.

Step 5: Calculate the True Positive Rate (TPR). This measures how many real defects we caught. We divide TP by the sum of TP and FN.
TPR = 7 divided by (7 + 3) = 7 divided by 10 = 0.70.
The model found 70% of the defective parts.

Step 6: Calculate the False Positive Rate (FPR). This measures how often we falsely alarm on good parts. We divide FP by the sum of FP and TN.
FPR = 50 divided by (50 + 9,940) = 50 divided by 10,000 = 0.005.
The model falsely alarms on only 0.5% of good parts.

Step 7: Calculate Precision. This measures how many of our alarms are real. We divide TP by the sum of TP and FP.
Precision = 7 divided by (7 + 50) = 7 divided by 57 ≈ 0.123.
When the model says 'defective', it is correct only about 12.3% of the time. This means roughly 88% of the alarms are false alarms.

Step 8: Calculate Accuracy for comparison. We divide the total correct predictions (TP + TN) by the total number of parts.
Accuracy = (7 + 9,940) divided by 10,000 = 9,947 divided by 10,000 = 0.9947.
The accuracy is 99.47%. This looks excellent, but it hides the fact that we missed 30% of the defects and generated many false alarms. TPR, FPR, and Precision reveal the true performance on the rare event.


## From scratch, in Python

```python
# We will compute the four counts from a list of true outcomes and model predictions.
# A list in Python is an ordered collection of values, written in square brackets.
# Here, 1 means 'rare event', 0 means 'no event'.

true = [0, 0, 1, 0, 0, 0, 0, 0, 1, 0]  # only two defects out of ten
predicted = [0, 1, 1, 0, 0, 0, 0, 0, 0, 0]  # model flags the second item incorrectly

# Initialize counters to zero.
tp = 0  # true positive
fp = 0  # false positive
tn = 0  # true negative
fn = 0  # false negative

# Loop through each pair of true and predicted values.
# for i in range(len(true)): visits index 0, 1, ..., 9 in order.
for i in range(len(true)):
    t = true[i]   # actual outcome (0 or 1)
    p = predicted[i]  # model's prediction (0 or 1)
    if t == 1 and p == 1:
        tp = tp + 1
    elif t == 1 and p == 0:
        fn = fn + 1
    elif t == 0 and p == 1:
        fp = fp + 1
    else:
        tn = tn + 1

# Print the counts
print("TP:", tp, "FP:", fp, "TN:", tn, "FN:", fn)

# Compute rates as floating-point numbers (division of integers in Python 3 gives a float)
tpr = tp / (tp + fn) if (tp + fn) > 0 else 0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
precision = tp / (tp + fp) if (tp + fp) > 0 else 0

print("True Positive Rate:", tpr)
print("False Positive Rate:", fpr)
print("Precision:", precision)
```


## What the code does

First, we define two lists: `true` holds the actual outcomes, `predicted` holds what our model guessed. A list is a collection of values in a specific order, starting at index 0. We then set four counters to zero. The `for` loop goes through each position in the lists, one at a time. Inside the loop, we pull the actual value `t` and the predicted value `p`. Then we use `if` and `elif` to check the four possibilities. If both are 1, it's a true positive; if the true is 1 but predicted is 0, it's a false negative; if true is 0 but predicted is 1, it's a false positive; otherwise both are 0, so it's a true negative. We add one to the corresponding counter. After the loop, we compute the rates by dividing. The `if ... else` in the rate lines guards against division by zero if there are no positives at all. The printed results tell us how well the model performs on the rare class. In this tiny example, we see the model catches one of the two defects (TPR=0.5) and flags one non-defect (FPR=0.125) — this is the trade-off we care about.


## Common misconceptions

**Accuracy is the right metric** — My model is 99.5% accurate, so it's excellent.

Accuracy counts how many predictions are correct. When the rare class occurs only 0.1% of the time, a model that always predicts 'not event' is 99.9% accurate but never catches any event. It is useless for the task. In fact: Accuracy hides the performance on the rare class. You must look at true positive rate, false positive rate, and precision to understand if the model actually finds the events you care about.

**Threshold is a fixed technical constant** — The threshold should be set to 0.5 because that is what the model outputs.

A probability of 0.5 is not a magic value. The optimal threshold depends on the costs of false positives vs false negatives, which are business decisions, not model decisions. In fact: The threshold should be tuned by evaluating the cost of each kind of error and choosing the point that minimizes total expected cost. This is often done by looking at a precision-recall curve.

**A high true positive rate is always best** — I want to catch every rare event, so I should make the model as sensitive as possible.

Pushing the threshold down to catch more true positives will almost always increase false positives. If false positives are expensive (e.g., declining legitimate transactions), you may end up losing more money than you save from catching fraud. In fact: The best threshold balances the cost of missing an event against the cost of a false alarm. In some domains (e.g., cancer screening), a high false positive rate is acceptable because of the high cost of missing a case, but in others it is not.

**Rare events are irrelevant if you have lots of data** — If I have a million examples, the model will learn to predict the rare event well even if it happens only once.

Even with huge data, the rare class has very few examples. The model naturally learns to predict the common class to minimize average error. Without special handling (class weights, resampling, threshold adjustment), it will ignore the rare class. In fact: Rare event prediction requires explicit techniques: oversampling the rare class, undersampling the common class, or weighting the loss function. The imbalance is the core challenge, not just a nuisance.


## In short

Rare event prediction is about balancing two kinds of mistakes: missing the rare event (false negative) and crying wolf (false positive). Accuracy is useless because predicting the common class scores well while doing nothing. You must compute true positive rate, false positive rate, and precision, then choose a decision threshold based on the costs of each error. This pattern appears across every area of AI, from LLM safety filters to fraud detection.


## Where this is used

Rare event prediction is everywhere in modern AI, often hidden behind other names.

**In large language models (LLMs):** LLMs are trained on billions of words, and some important patterns are rare — for example, a specific factual error or a toxic phrase. When training, the loss function treats every token equally, but the model can learn to output common phrases while ignoring rare but important ones. To handle this, practitioners often use **class weighting** — giving more weight to examples of the rare class in the loss function. In terms of our trade-off, this shifts the threshold to catch more rare events (like harmful output) at the cost of more false positives (flagging benign text as harmful). For example, safety filters in LLMs are deliberately set with high sensitivity (low threshold) to catch dangerous content, even if it means occasionally blocking harmless requests. Developers who build such filters literally write code to adjust the threshold after the model outputs a probability score, choosing a value that balances missed harms against user annoyance.

**In small models running on devices (like phones):** Keyword spotting (e.g., 'Hey Siri') is a rare event: the phrase occurs only occasionally in a continuous audio stream. The model runs constantly on a low-power chip, and it must not miss the keyword (a false negative is infuriating) but also must not wake up on random speech (a false positive drains battery). In practice, the developer sets a threshold on the model's confidence score. If the score is above, say, 0.3, the device wakes up; if below, it ignores. The threshold is tuned on real usage data, often balancing two costs: the cost of missing a wake word (user frustration) versus the cost of battery drain and false activations. The same idea applies to detecting 'OK Google' or other trigger phrases.

**In retrieval-augmented generation (RAG):** RAG systems retrieve relevant documents from a large corpus to help an LLM answer questions. A crucial step is **relevance filtering**: deciding whether a retrieved document is actually relevant to the query. Relevant documents are rare compared to the huge pool of potential matches. If the filter is too strict (high threshold), the model may miss important context (false negatives), leading to wrong answers. If too loose, it includes irrelevant noise (false positives), confusing the generation. Developers often use a classifier or a simple similarity score with a threshold that is tuned on a validation set, measuring RAG-specific metrics like answer accuracy and faithfulness. The precision–recall trade-off directly influences the quality of the final answer.

**In agentic systems that plan and call tools:** Agents that call external tools (like a web search or a calculator) must decide when to act. An agent that searches too often wastes time and money; one that never searches misses critical information. Deciding whether to perform a tool call is a rare event decision if the agent's default path is to answer from memory. The agent's policy is trained or manually tuned with thresholds that balance the cost of an unnecessary call (latency, cost) against the cost of a missing call (incorrect answer). In practice, this is often done via reinforcement learning, but in simpler systems a developer writes a rule: 'search only if the model's confidence is below 0.7'. This is exactly a threshold choice.

**In everyday applied machine learning:** Fraud detection, manufacturing defect detection, disease screening, and predictive maintenance are all classic rare event problems. In fraud detection, a credit card transaction is fraudulent only 0.1% of the time. A model that always says 'not fraud' is 99.9% accurate but catches nothing. The bank must decide how many false alarms (declined legitimate transactions) they can tolerate compared to missed fraud. They use precision and recall curves to pick a threshold that maximizes profit, considering the cost of a fraudulent transaction vs. the cost of a declined customer. In disease screening, a test for a rare disease is often set to be highly sensitive (catch all possible cases) because the cost of missing a case is high, and then a follow-up, more specific test is used to reduce false positives. In each case, the developer writes code to compute TP/FP/TN/FN from validation data, plots the trade-off curve, and chooses a threshold based on business costs. The underlying mathematics is identical to what you just did by hand.


---

Read this on the web: https://basavapurushottam.com/learn-ai/rare-event-prediction
