# Naive Bayes

A probabilistic classifier applying Bayes' rule under the assumption that features are conditionally independent given the class.

**Rests on:** Bayesian Inference and Decision Theory, Fitting Distributions

**Leads to:** Fake News Detection

*Machine Learning · Machine Learning Foundations*


## Why learn this

You want to make better guesses when you don't have complete information. Naive Bayes is a simple method for turning a list of observed facts into a clear probability—like estimating the chance that an email is spam based on specific words in it. It is one of the fastest ways to build a working model for tasks like sorting messages or judging opinions, and it gives you results you can tune if some mistakes are more expensive than others.


## The intuition

Imagine you are a doctor with a patient who has a fever, a cough, and a runny nose. You want to guess whether they have the flu or just a cold. You have seen thousands of patients before, so you know: among all your patients, roughly 30% have the flu and 70% have a cold. This is your starting belief, called the base rate. You also know that among flu patients, 80% have a fever, 60% have a cough, and 40% have a runny nose. Among cold patients, 20% have a fever, 30% have a cough, and 70% have a runny nose. Now a new patient walks in with all three symptoms. You don't know the exact combination from your records — you only have the single-symptom rates. So you make a leap: you assume that the symptoms appear independently of each other given the disease. That means you can just multiply the probabilities: for flu, 0.8 × 0.6 × 0.4 = 0.192; for cold, 0.2 × 0.3 × 0.7 = 0.042. You also multiply by the overall rates: for flu, 0.192 × 0.3 = 0.0576; for cold, 0.042 × 0.7 = 0.0294. At this stage, these numbers (0.0576 and 0.0294) are not yet probabilities because they do not add up to 1 (or 100%). To fix this, you normalize them by dividing each by the sum of all possible outcomes (0.087). This rescales them so they fit together perfectly: flu is about 66%, cold is about 34%. So you lean flu. That is Naive Bayes: use single-feature rates, multiply them together, adjust for how common each class is, then normalize so the answers add up to 100%. The 'naive' part is the independence assumption — you know it's usually false, because symptoms often travel together, but it makes the arithmetic possible and it usually still gives a useful answer.


## An analogy

Think of a witness in a courtroom. The witness is trying to decide whether the defendant is guilty or innocent based on three pieces of evidence: a fingerprint, an alibi, and a motive. The witness does not have a perfect record of how often those three appear together for guilty and innocent people. Instead, they have separate statistics: of the last 100 guilty verdicts, 70 had fingerprints, 50 had weak alibis, and 60 had motives. Of the last 100 innocent verdicts, 10 had fingerprints, 30 had weak alibis, and 40 had motives. Also, overall, 20% of defendants are guilty and 80% are innocent. Now a new case has all three pieces of evidence. The witness assumes those pieces are independent given guilt or innocence — that having a fingerprint does not make a weak alibi more likely, and so on. This is usually false: criminals who leave fingerprints might also be sloppier with their alibis. But the witness goes ahead because the separate statistics are all they have. They calculate: for guilty, 0.7 × 0.5 × 0.6 × 0.2 = 0.042; for innocent, 0.1 × 0.3 × 0.4 × 0.8 = 0.0096. Normalizing (0.042 + 0.0096 = 0.0516), they get guilty ≈ 81% and innocent ≈ 19%. The witness issues a verdict of 'probably guilty' with a confidence level. Where does the analogy break down? In real life, evidence is often strongly correlated — a weak alibi is more likely if there is a motive — and the witness's assumption of independence can bias the final probability.


## Definition

Naive Bayes is a method for classifying items by multiplying the likelihood of their features appearing in each category together, then scaling the results so they represent complete probabilities.


## How it connects

This sits beside K-Nearest Neighbours and Decision Trees as one of the 'classical' algorithms — the ones that are small, fast, and easy to understand. It builds directly on the ideas of Data Mining (finding patterns in large data) and leads naturally to Logistic Regression, which learns the weights from data rather than counting frequencies.


## The mathematics

This concept rests on a branch of mathematics called probability. Probability is the mathematics of uncertainty — it gives us a way to assign numbers between 0 and 1 to 'how likely' something is. You already use it informally: 'there's a 70% chance of rain' or 'I'm 80% sure I left the keys on the table.' The specific operation Naive Bayes uses is multiplication of probabilities to combine evidence.

The rule is Bayes' rule, which answers the question: 'Given that I see these symptoms, how likely is each possible disease?' It is called 'Bayes' rule' after Thomas Bayes, an 18th-century minister. The rule says: the probability of a class given the evidence is proportional to the probability of the evidence given the class times the probability of the class.

In other words, you start with what you know about the class in general (how common it is), then multiply by how strongly each piece of evidence points to that class. Because you assume the features are independent given the class, you can multiply the individual conditional probabilities instead of needing the joint distribution, which would be impossible to estimate from limited data.

The formula for this calculation is:

P(C | F₁, F₂, ..., Fₙ) ∝ P(C) × ∏ᵢ P(Fᵢ | C)

Here is what each part of that formula means in words:

1. P(C | F₁, F₂, ..., Fₙ): This is the probability of the class C, given that we have observed features F₁ through Fₙ. In our email example, this is the final answer: 'How likely is this email to be spam, now that I see the words "free" and "win"?'

2. The symbol ∝ (read as 'is proportional to'): This means the left side equals the right side multiplied by some constant number. We do not need to know that constant number yet because we will compare the results for different classes later.

3. P(C): This is the prior probability of class C. It is how common class C is in the entire dataset, before looking at any specific features. In our example, this is the base rate of spam emails.

4. The symbol ×: This means multiply.

5. The symbol ∏ᵢ (read as 'the product over i'): This tells you to take the probability for each feature Fᵢ and multiply them all together. If there are three features, you multiply P(F₁|C) by P(F₂|C) by P(F₃|C).

6. P(Fᵢ | C): This is the conditional probability of feature Fᵢ given class C. It answers: 'If I know this email is spam, how likely is it to contain word Fᵢ?'

The key assumption in Naive Bayes is that the features are independent given the class. This allows us to use step 5 (multiplying individual probabilities) instead of trying to calculate the probability of all features appearing together at once, which would require much more data.


## A worked example

We have a simple dataset of 10 emails: 4 are spam (S), 6 are not (NS). Among the spam emails, 3 contain the word 'free', 2 contain 'win', and 1 contains 'meeting'. Among non-spam, 1 contains 'free', 1 contains 'win', and 5 contain 'meeting'. A new email contains both 'free' and 'win'. We will calculate the probability it is spam.

Step 1: Calculate the prior probabilities (how common each class is in the whole dataset).
P(S) = 4/10 = 0.4. This means 4 out of 10 emails are spam.
P(NS) = 6/10 = 0.6. This means 6 out of 10 emails are not spam.

Step 2: Calculate the conditional probabilities for each feature in each class.
For Spam (S):
P('free' | S) = 3/4 = 0.75. Of the 4 spam emails, 3 contain 'free'.
P('win' | S) = 2/4 = 0.5. Of the 4 spam emails, 2 contain 'win'.

For Non-Spam (NS):
P('free' | NS) = 1/6 ≈ 0.167. Of the 6 non-spam emails, 1 contains 'free'.
P('win' | NS) = 1/6 ≈ 0.167. Of the 6 non-spam emails, 1 contains 'win'.

Step 3: Combine the evidence for each class using the Naive Bayes formula.
The formula says: Score = Prior × Product of Conditional Probabilities.

For Spam:
Score(S) = P(S) × P('free' | S) × P('win' | S)
Score(S) = 0.4 × 0.75 × 0.5
Score(S) = 0.3 × 0.5
Score(S) = 0.15

For Non-Spam:
Score(NS) = P(NS) × P('free' | NS) × P('win' | NS)
Score(NS) = 0.6 × 0.167 × 0.167
Score(NS) ≈ 0.6 × 0.028
Score(NS) ≈ 0.017

Step 4: Normalize the scores to get actual probabilities.
The scores 0.15 and 0.017 are proportional to the true probabilities, but they do not add up to 1 yet. To fix this, we divide each score by the sum of all scores.

Total Score = 0.15 + 0.017 = 0.167

Probability(S | 'free', 'win') = Score(S) / Total Score
Probability(S | 'free', 'win') = 0.15 / 0.167 ≈ 0.898

Probability(NS | 'free', 'win') = Score(NS) / Total Score
Probability(NS | 'free', 'win') = 0.017 / 0.167 ≈ 0.102

Result: The email is about 89.8% likely to be spam, and about 10.2% likely to be non-spam. So we classify it as spam.


## From scratch, in Python

```python
# We'll build a Naive Bayes classifier from scratch.
# First, our dataset: each email is a list of words.

# The email is represented as a list of words.
# In Python, a list is an ordered collection of items, written inside square brackets.
# For example, ['free', 'win'] is a list of two strings.

# Our training data:
# We have 10 emails. We'll represent each email as a dictionary mapping word to count.
# A dictionary in Python is like a real dictionary: you look up a key (a word) and get a value (the count).
# For simplicity, we'll count each occurrence.

emails = [
    {'free': 1, 'win': 0, 'meeting': 0},  # spam
    {'free': 0, 'win': 1, 'meeting': 0},  # spam
    {'free': 1, 'win': 1, 'meeting': 0},  # spam
    {'free': 1, 'win': 0, 'meeting': 1},  # spam
    {'free': 0, 'win': 0, 'meeting': 1},  # not spam
    {'free': 0, 'win': 0, 'meeting': 1},  # not spam
    {'free': 0, 'win': 0, 'meeting': 1},  # not spam
    {'free': 0, 'win': 0, 'meeting': 1},  # not spam
    {'free': 1, 'win': 0, 'meeting': 1},  # not spam
    {'free': 0, 'win': 1, 'meeting': 1},  # not spam
]
# The labels: 1 = spam, 0 = not spam.
labels = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0]

# Our vocabulary: all unique words that appear.
vocab = ['free', 'win', 'meeting']

# Step 1: compute the prior probabilities P(spam) and P(not spam).
# We count how many emails are spam and divide by total.
num_spam = sum(labels)  # sum of labels gives number of 1s
num_not = len(labels) - num_spam
total = len(labels)

p_spam = num_spam / total
p_not = num_not / total

# Step 2: compute the conditional probabilities P(word | class).
# We'll count for each class how many emails contain that word.
# We'll store these in a dictionary: {class: {word: count}}
# A dictionary in Python holds key-value pairs. Here the key is the class (0 or 1)
# and the value is another dictionary mapping word to count.

word_counts = {0: {'free': 0, 'win': 0, 'meeting': 0},
               1: {'free': 0, 'win': 0, 'meeting': 0}}

# For each email and its label, we add its word counts to the corresponding class.
for email, label in zip(emails, labels):
    # zip pairs up the two lists element-wise.
    for word in vocab:
        word_counts[label][word] += email[word]

# Now convert counts to probabilities by dividing by the number of emails in each class.
# We'll use a 'smoothing' trick: add 1 to every count to avoid probabilities of zero.
# This is called Laplace smoothing, and it prevents a single unseen word from zeroing everything.

cond_probs = {0: {}, 1: {}}
for label in [0, 1]:
    # number of emails of this class
    class_size = num_spam if label == 1 else num_not
    # we add 1 to the numerator and number of possible words to the denominator
    denominator = class_size + len(vocab)
    for word in vocab:
        # count plus 1 (smoothing) divided by denominator
        cond_probs[label][word] = (word_counts[label][word] + 1) / denominator

# Step 3: classify a new email.
# New email: contains 'free' and 'win', not 'meeting'.
new_email = {'free': 1, 'win': 1, 'meeting': 0}

# We compute the unnormalized probabilities for each class.
# Start with the prior probability.
score_spam = p_spam
score_not = p_not

# Multiply by each word's conditional probability if the word is present,
# and by (1 - conditional) if the word is absent.
for word in vocab:
    if new_email[word] == 1:
        score_spam *= cond_probs[1][word]
        score_not *= cond_probs[0][word]
    else:
        # if word is absent, we use 1 - probability
        score_spam *= (1 - cond_probs[1][word])
        score_not *= (1 - cond_probs[0][word])

# Step 4: normalize so the two scores sum to 1.
total_score = score_spam + score_not
p_spam_given = score_spam / total_score
p_not_given = score_not / total_score

print(f"P(spam | email) = {p_spam_given:.4f}")
print(f"P(not spam | email) = {p_not_given:.4f}")

# The output will be something like:
# P(spam | email) = 0.8571
# P(not spam | email) = 0.1429
```


## The same thing, with a framework

In practice, you would rarely implement Naive Bayes by hand; you would use scikit-learn. Here is the same thing with `MultinomialNB`:

```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

# Example text data
texts = [
    'free offer win now',
    'win a free prize',
    'meeting agenda tomorrow',
    'free free free',
    'win win win',
    'meeting at 3pm'
]
labels = [1, 1, 0, 1, 1, 0]  # 1=spam, 0=not

# Convert text to word counts: each email becomes a vector of counts.
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Train the Naive Bayes classifier.
clf = MultinomialNB()
clf.fit(X, labels)

# New email
new_text = ['free win']
X_new = vectorizer.transform(new_text)
proba = clf.predict_proba(X_new)
print(proba)  # e.g., [[0.12, 0.88]] meaning 12% not spam, 88% spam
```

`CountVectorizer` does the same counting and dictionary building you did manually; `MultinomialNB` applies the same multiplication and smoothing under the hood.


## What the code does

The code builds a simple Naive Bayes classifier from a small dataset. We first define a list of emails, each represented as a dictionary mapping words to counts (0 or 1 here). The labels list tells us which are spam (1) and which are not (0). We then compute the prior probabilities by counting how many emails are spam and not spam. Next, we loop over each email to count how many times each word appears in each class, storing those counts in a dictionary. We convert counts to conditional probabilities using Laplace smoothing: we add 1 to every count and add the number of words to the denominator, so no probability is ever zero. To classify a new email, we start with the prior probabilities and multiply by the conditional probabilities for each word that is present, and by (1 − conditional) for each word that is absent (that is the 'feature absent' case). Finally, we normalize the two scores to get probabilities that sum to 1. The output shows the probability the email is spam.


## Common misconceptions

**The independence assumption is always true.** — Naive Bayes assumes features are independent, so it must only be used when they actually are.

The assumption is almost never true in practice — words in a sentence are correlated, symptoms co-occur, and so on. Yet Naive Bayes still performs surprisingly well. The reason is that for classification, we only need the relative ordering of probabilities, not exact probabilities. So even if the probabilities are off, the ranking often is not. In fact: The assumption is false by design; the classifier works because classification often only needs approximate ordering. It is 'naive' because it ignores correlation, but that simplification gets you speed and low data requirements.

**It's always better than a simple heuristic.** — Naive Bayes is a sophisticated algorithm, so it must beat any rule-of-thumb.

For some problems, a simple rule (like 'if it contains the word free, mark as spam') can be just as good and far more interpretable. Naive Bayes only wins when the patterns are not that simple. In fact: Naive Bayes is a baseline — it is often better than random, but a carefully crafted rule or a more complex model may beat it. It is always worth comparing it against a dummy baseline first.

**The output is the true probability** — If Naive Bayes says 90% chance of spam, that email is indeed 90% likely to be spam.

Because the independence assumption is wrong, the final number is biased. It may be overconfident (or underconfident). The probability is a useful ranking score, but not a calibrated probability. In fact: Treat the output as a confidence score for ranking, not as a true probability. If you need reliable probabilities, apply calibration techniques (like Platt scaling) after training.

**Naive Bayes is only for text** — You can only use Naive Bayes on word counts.

Naive Bayes works with any categorical features — image pixel histograms, sensor readings (binned into ranges), transaction amounts (binned) — as long as you estimate the conditional probabilities. In fact: There is a Gaussian variant for continuous features and a Bernoulli variant for binary features. It is a general classifier, not a text-only tool.

**Zero probabilities are fine** — If a word never appears in spam, its probability is 0.

A single zero makes the whole product zero, so any email containing that word would be classified as not spam just because the training data happened not to include it. In fact: Use Laplace smoothing (add 1 to all counts) to avoid zero probabilities. This ensures no single unseen word can zero out the calculation.


## In short

Naive Bayes is a probabilistic classifier that multiplies the probabilities of each feature given the class to estimate which class is most likely. It assumes features are independent (the 'naive' part), which is often wrong but rarely fatal. It is fast, needs little data, and works well for text classification, spam filtering, and as a strong baseline. Remember: the output is a ranking score, not a calibrated probability, and you must use smoothing to avoid zeros.


## Where this is used

Naive Bayes is used in many real-world AI systems, and its role depends on the scale of the system.

**Large language models (LLMs) and modern NLP.** Naive Bayes is not the engine of GPT-4 or ChatGPT — those use deep neural networks with attention. But Naive Bayes appears in the *pre-processing* and *benchmark* stages. For example, you might use it to build a fast baseline for sentiment classification of a large text corpus before training a deep model. It is also used in the *tokenization* step: some tokenizers (like BPE) use a byte-pair encoding that is a form of frequency counting, and Naive Bayes can be used to rank candidate token splits.

**Small language models on devices.** When you need a model that runs on a phone or a Raspberry Pi, a full neural network is often too big and too slow. Naive Bayes is the classic choice for on-device spam filtering, simple language detection, and the 'smart reply' suggestions on older keyboards. The model is tiny — just a table of word probabilities per class — and can be updated locally as the user corrects it. This is a real, deployed use of the algorithm.

**Retrieval-augmented generation (RAG).** RAG pipelines retrieve relevant documents to feed to an LLM. The retrieval step often uses a *classifier* to decide which documents are worth retrieving. While the actual semantic search uses embeddings, Naive Bayes can be used as a fast filter to pre-select a small set of candidates before the expensive embedding comparison. For example, in a legal document search, you might have a Naive Bayes model that classifies documents as 'relevant' or 'not relevant' based on the presence of key legal phrases, and then only run the embedding similarity on the 'relevant' ones.

**Agentic systems.** An agent that plans and calls tools needs to decide, at each step, which tool to use next. Naive Bayes can be the *tool selector* when the set of tools is small and the input is text. For instance, a voice assistant might use a Naive Bayes classifier to decide whether the user is asking about the weather, setting a timer, or playing music. The input voice is transcribed to text, and the classifier picks the most likely intent. It is not the modern deep-learning way, but it works and is very fast.

**Everyday applied machine learning.** This is where Naive Bayes shines the most and is still heavily used:

- **Spam filtering** (as in our example): every major email provider has a Naive Bayes component in its spam filter, often combined with a deep model.
- **Sentiment analysis**: for small to medium text data, Naive Bayes is a strong and interpretable baseline. You can see exactly which words push the prediction to positive or negative.
- **Churn prediction** (which you have in your library): Naive Bayes can model the probability that a customer will leave, given features like usage frequency, support calls, and plan type. Its output is a probability, so you can set a threshold for who to call — exactly what a retention team needs.
- **Crop disease detection** (also in your library): though usually done with convolutional networks, a simple Naive Bayes on colour histograms can be a fast pre-screening tool on a low-power sensor, flagging likely diseased leaves for closer inspection.
- **Recommendation systems**: a Naive Bayes classifier can predict what product a user is likely to buy based on the presence of certain tags or viewed categories, especially in cold-start scenarios where there is no user history yet.
- **Forecasting**: rare but possible — for binary outcomes (e.g., will it rain tomorrow?), Naive Bayes on categorical weather features works as a simple baseline.
- **Fraud detection**: credit card companies use layered models; Naive Bayes can run in real time on a transaction to score the probability of fraud before the full deep model runs, and if the probability is very high, it triggers an immediate hold.
- **Speech recognition**: the acoustic model that converts sound to phonemes is often a neural net, but the *language model* that decides the most likely word sequence can be a Naive Bayes n-gram language model in a small embedded system.

When a developer reaches for a library, they will use scikit-learn's `MultinomialNB` for text data, `BernoulliNB` for binary features, or `GaussianNB` for continuous features. The developer's main job is to convert their data into a format the classifier expects — usually a count vector or a feature table — and then to evaluate whether the output probabilities are well-calibrated (i.e., whether a predicted 70% chance is right about 70% of the time). What breaks when it is done badly is: using Naive Bayes on highly correlated features (like duplicated text), failing to smooth the probabilities so one zero kills the calculation, or assuming the output is a true probability when the independence assumption distorts it.


---

Read this on the web: https://basavapurushottam.com/learn-ai/naive-bayes
