# Bayesian Inference and Decision Theory

Updating beliefs with Bayes' rule and then choosing the action that minimises expected loss under the resulting posterior.

**Rests on:** Fitting Distributions

**Leads to:** ELBO, Fisher Information, Gaussian Process Regression, Naive Bayes

*Mathematical Foundations · Bayesian Methods*


## Why learn this

You make decisions every day under uncertainty. Do you take an umbrella? Do you approve a loan application? Does this patient have a serious condition? In professional life, these are not guesses; they are calculations based on incomplete information. This topic teaches you how to update your beliefs rationally as new facts arrive, and how to choose actions that minimize regret. It is the framework behind systems that diagnose diseases, filter spam, drive cars, and recommend products. Understanding this helps you evaluate how these systems work and how to trust (or question) their outputs.


## The intuition

Imagine you need to decide whether a specific email is spam. Before you even open it, you have an initial guess based on your experience — perhaps 20% chance it is junk because your inbox gets flooded occasionally. This starting estimate is what we call a "prior belief" (the word "prior" just means "before").

Then you read the subject line: "URGENT: CLAIM YOUR PRIZE NOW!!!". This is new evidence. You know that legitimate messages rarely use all caps and excessive punctuation. So, this evidence suggests the email is almost certainly spam. You update your guess dramatically, say to 95% probability of being spam. This updated belief is called the "posterior belief" (the word "posterior" means "after"). The math that lets you combine your starting guess with the new evidence to get the updated guess is known as "Bayes' rule".

But knowing the probability isn't enough. You have to act. If you delete the email, two things can happen:
1. It was spam (good outcome: no wasted time).
2. It was important (bad outcome: you missed a critical message).

If you keep it, two other things can happen:
1. It was spam (annoying: you spent time reading junk).
2. It was important (good outcome: you handled a critical matter).

Which action is best? This is where "decision theory" comes in. It asks you to weigh the costs. Let's assign values (losses):
- Deleting real mail: Loss = 100 (missing important info).
- Keeping spam: Loss = 5 (minor annoyance).

If you believe there is a 95% chance it is spam:
- Expected loss of deleting = (5% chance it's real * 100) + (95% chance it's spam * 0) = 5.
- Expected loss of keeping = (5% chance it's real * 0) + (95% chance it's spam * 5) = 4.75.

Here, keeping the email has a lower "expected loss". The expected loss is simply the average cost you would suffer if you made this decision many times under identical conditions. So, rationally, you keep it.

Bayes' rule updates your belief; decision theory helps you act on that belief by calculating which choice minimizes your average future regret.


## An analogy

Think of a chef tasting a soup to decide if it needs more salt. The chef has a prior belief about how salty the recipe should be (based on experience). Then the chef tastes the soup (the evidence) and updates their belief about how much salt is actually in there (the posterior belief). Now the chef must decide: add salt, or stop cooking? If they add too much, the soup is ruined (high loss). If they add too little, it is bland but edible (moderate loss). The chef weighs the probability that the soup is currently under-salted against the cost of over-salting it. They choose the action (adding a pinch or stopping) that minimizes the expected disappointment. In this analogy, the "taste" is the data, the "recipe knowledge" is the prior belief, and the "decision to add salt" is the action chosen via decision theory.


## Definition

Bayesian inference is the method of updating an initial probability estimate (prior) with new evidence to produce a revised estimate (posterior) using Bayes' rule; decision theory is the process of selecting the optimal action by choosing the option that results in the lowest expected loss, calculated from these updated probabilities.


## How it connects

This concept sits beside "Probability Distributions," which are just mathematical ways to describe how likely different outcomes are, and "Statistics," which is the broader study of collecting and analyzing data. While Probability Distributions give you the language to describe uncertainty, Bayesian inference gives you the tool to update that description as you learn more. Decision theory then uses those updated descriptions to guide action.


## The mathematics

This topic combines two ideas: how to update beliefs when new information arrives (Bayesian inference), and how to choose the best action when outcomes are uncertain (decision theory).

Part 1: Updating Beliefs with Bayes' Rule

Imagine you have a guess about whether something is true. In Bayesian terms, this initial guess is called the **prior**. When you see new evidence, you update your guess. The updated guess is called the **posterior**.

Bayes' rule tells us exactly how to do this update. It says:

Posterior = (Likelihood × Prior) / Evidence

Let's define each word in that formula:

1. **Prior**: Your belief before seeing the new evidence. For example, if you are guessing whether a coin is fair, your prior might be "I think there is a 50% chance it is fair."
2. **Likelihood**: How likely the new evidence is, assuming your current belief is correct. If you believe the coin is fair, what is the chance of seeing the specific pattern of heads and tails you just observed?
3. **Evidence**: The total probability of seeing that evidence under *any* possible belief. This number ensures that all your probabilities add up to 100%. It acts as a normalizing constant.
4. **Posterior**: Your new, updated belief after combining the prior and the likelihood.

The formula is often written as:
P(Hypothesis | Evidence) = [P(Evidence | Hypothesis) × P(Hypothesis)] / P(Evidence)

Here is what the notation means in plain English:
- "P(A | B)" is read as "the probability of A given B." It asks: If we know B is true, how likely is A?
- The vertical bar "|" means "given."
- The numerator "P(Evidence | Hypothesis) × P(Hypothesis)" represents the joint chance that both the hypothesis and the evidence are true.
- The denominator "P(Evidence)" scales this number so it becomes a valid probability between 0 and 1.

Part 2: Making Decisions with Expected Loss

Once you have updated your beliefs (the posterior), you need to decide what to do. Decision theory provides a method for choosing the best action by looking at the consequences.

Every action has potential outcomes, and each outcome has a cost or **loss**. For example, if you are a doctor deciding whether to treat a patient, the loss might be the harm caused by treating a healthy person (side effects) versus the harm of not treating a sick person (death).

To choose the best action, we calculate the **expected loss** for each possible action. "Expected" here means a weighted average. We weigh each possible loss by how likely that outcome is (using our posterior probabilities).

The formula for expected loss is:

Expected Loss = Sum over all outcomes [ (Loss of taking this action if this outcome is true) × (Probability of this outcome being true) ]

In mathematical notation, this looks like:
E[L(a)] = Σ_x L(a, x) * P(x | data)

Let's break down the symbols:
- "E" stands for Expectation. It means we are calculating an average.
- "L(a, x)" is the Loss function. It tells you the cost of taking action 'a' when the true state of the world is 'x'.
- "P(x | data)" is the posterior probability that the true state is 'x', given the data we have observed.
- "Σ_x" (Sigma x) means "sum over all possible values of x." You list every possible outcome, multiply its loss by its probability, and add them all up.

The best action is the one with the smallest expected loss. This is because it minimizes the average cost you would pay if you had to make this decision many times under the same uncertainty.

Summary

Bayesian inference gives us the probabilities (the posterior) of different states of the world after seeing data. Decision theory uses those probabilities to calculate the average cost (expected loss) of each possible action. The optimal decision is the action with the lowest expected loss.


## A worked example

Let's walk through a concrete example. We will first use Bayes' rule to update our belief about a disease, and then use decision theory to decide whether to treat the patient.

Scenario: A Medical Test

You are a doctor. A patient comes in for a test for Disease D.

1. The Prior (Before the test)
   - The disease is rare. In the general population, only 1% of people have it.
   - So, P(Disease) = 0.01.
   - Consequently, P(No Disease) = 0.99.

2. The Test Characteristics (The Likelihoods)
   - The test is not perfect.
   - Sensitivity: If a person has the disease, the test correctly says "Positive" 95% of the time. So, P(Positive | Disease) = 0.95.
   - Specificity: If a person does *not* have the disease, the test correctly says "Negative" 90% of the time. This means it incorrectly says "Positive" 10% of the time. So, P(Positive | No Disease) = 0.10.

3. The Observation
   - The patient takes the test, and the result is Positive.

Step 1: Calculate the Posterior Probability (Bayesian Inference)

We want to find P(Disease | Positive). This is the probability that the patient has the disease *given* that they tested positive.

Using Bayes' rule:
P(Disease | Positive) = [P(Positive | Disease) × P(Disease)] / P(Positive)

First, we need the denominator, P(Positive). This is the total probability of getting a positive result, regardless of whether the patient has the disease or not. We calculate this by adding two scenarios:
- Scenario A: The patient has the disease AND tests positive.
- Scenario B: The patient does not have the disease AND tests positive (a false positive).

P(Positive) = [P(Positive | Disease) × P(Disease)] + [P(Positive | No Disease) × P(No Disease)]
P(Positive) = (0.95 × 0.01) + (0.10 × 0.99)
P(Positive) = 0.0095 + 0.099
P(Positive) = 0.1085

Now we can calculate the posterior:
P(Disease | Positive) = (0.95 × 0.01) / 0.1085
P(Disease | Positive) = 0.0095 / 0.1085
P(Disease | Positive) ≈ 0.0876

So, even though the test was positive, there is only about an 8.76% chance the patient actually has the disease. This seems low, but it is because the disease is very rare (the prior is small), so false positives outnumber true positives.

Step 2: Calculate Expected Loss (Decision Theory)

Now you must decide: Treat the patient (Action T) or Do Not Treat (Action NT)?

Let's assign arbitrary loss values to illustrate the calculation:
- If you treat a healthy person, they suffer side effects. Let Loss(Treat | No Disease) = 10 units of harm.
- If you do not treat a sick person, they die. Let Loss(No Treat | Disease) = 100 units of harm.
- If you treat a sick person, they are cured. Loss(Treat | Disease) = 0.
- If you do not treat a healthy person, nothing happens. Loss(No Treat | No Disease) = 0.

We calculate the expected loss for each action using the posterior probabilities we just found:
P(Disease | Positive) ≈ 0.0876
P(No Disease | Positive) ≈ 1 - 0.0876 = 0.9124

Expected Loss for Treating (E[L(T)]):
= [Loss(Treat | Disease) × P(Disease | Positive)] + [Loss(Treat | No Disease) × P(No Disease | Positive)]
= (0 × 0.0876) + (10 × 0.9124)
= 0 + 9.124
= 9.124 units of harm.

Expected Loss for Not Treating (E[L(NT)]):
= [Loss(No Treat | Disease) × P(Disease | Positive)] + [Loss(No Treat | No Disease) × P(No Disease | Positive)]
= (100 × 0.0876) + (0 × 0.9124)
= 8.76 + 0
= 8.76 units of harm.

Comparison:
- Expected Loss of Treating: 9.124
- Expected Loss of Not Treating: 8.76

Since 8.76 is less than 9.124, the rational decision according to this model is **Not to Treat**.

Note: In a real medical context, you might change the loss values. If death is worth 1000 units of harm instead of 100, the expected loss of not treating would be 87.6, which is much higher than 9.124, and you would choose to treat. The math shows how sensitive the decision is to the assumed costs.


## From scratch, in Python

```python
# Plain Python implementation of Bayesian update and decision\n\n# Prior probability of disease\nprior = 0.01\n\n# Likelihood of positive test given disease / given no disease\nsensitivity = 0.95\nfalse_positive_rate = 0.10\n\n# Compute marginal probability of positive test\nprob_positive = sensitivity * prior + false_positive_rate * (1 - prior)\n\n# Posterior probability of disease given positive test (Bayes' rule)\nposterior = (sensitivity * prior) / prob_positive\n\nprint(f"Prior: {prior}")\nprint(f"Posterior after positive test: {posterior:.4f}")\n\n# Decision theory: choose action to minimise expected loss\n# Actions: treat or do nothing. Losses: if treat and disease -> 10 (cost of treatment), if treat and no disease -> 5 (unnecessary treatment side effects), if do nothing and disease -> 100 (serious), if do nothing and no disease -> 0.\n\n# define loss matrix: rows = action (0: treat, 1: do nothing), columns = true state (0: no disease, 1: disease)\nloss = [\n    [5, 10],   # treat\n    [0, 100]   # do nothing\n]\n\n# Expected loss for each action\n# For treat: loss if no disease * P(no disease) + loss if disease * P(disease)\nprob_no_disease = 1 - posterior\nexp_loss_treat = loss[0][0] * prob_no_disease + loss[0][1] * posterior\nexp_loss_do_nothing = loss[1][0] * prob_no_disease + loss[1][1] * posterior\n\nprint(f"Expected loss if treat: {exp_loss_treat}")\nprint(f"Expected loss if do nothing: {exp_loss_do_nothing}")\n\n# Choose the action with smaller expected loss\nif exp_loss_treat <= exp_loss_do_nothing:\n    best_action = "treat"\nelse:\n    best_action = "do nothing"\n\nprint(f"Best action: {best_action}")
```


## What the code does

This code first computes the posterior probability using Bayes' rule. It uses the prior (0.01) and the likelihoods (sensitivity 0.95, false positive rate 0.10) to find P(Pos), then computes P(D|Pos). The print statements show the prior and posterior.\n\nThen it moves to decision theory. The loss matrix is a list of lists: each inner list represents an action (treat or do nothing), and each element inside is the loss for each true state (no disease or disease). The expected loss for each action is the weighted sum of the losses, using the posterior probabilities as weights. For example, for treating, the expected loss is (loss if no disease) * (probability of no disease) + (loss if disease) * (probability of disease). The code calculates both expected losses and picks the action with the lower one.\n\nIn this example, the best action is likely to be 'do nothing' because the posterior is low (about 8.8%) and the loss of doing nothing when no disease is 0, while treating always incurs some cost.


## Common misconceptions

**A positive test means the disease is likely** — If I test positive for a rare disease, the probability I actually have it is close to 95%, because the test is 95% accurate.

This ignores the base rate (prior) of the disease. If the disease is rare, most positive tests are false positives. In fact: The posterior probability depends on the prior. A positive test can still result in a low posterior if the disease is rare. Always use Bayes' rule to combine the prior with the test's likelihood.

**The posterior is the same as the likelihood** — The probability of the data given the hypothesis is the same as the probability of the hypothesis given the data.

These are different conditional probabilities. P(data|hypothesis) is the likelihood; P(hypothesis|data) is the posterior. They are related by Bayes' rule but not equal. In fact: Use Bayes' rule to invert the conditioning. The prior is essential to this step.

**Decision theory just picks the most probable outcome** — If an action leads to a good outcome with the highest probability, it is the best action.

Probabilities alone are not enough; the consequences (losses) of each outcome matter. A low probability, high cost outcome can dominate the decision. In fact: Compute the expected loss for each action, weighting each outcome by its probability and its cost. Choose the action with the lowest expected loss.

**The prior is a personal opinion that has no place in science** — I should only use the data; the prior just introduces bias and should be ignored.

The prior encodes prior knowledge or assumptions. Without a prior, you cannot compute a posterior. Even a 'non-informative' prior is a modelling choice. In practice, priors help regularise and prevent overfitting. In fact: A prior is a modelling choice that must be defended. It is a strength of Bayesian methods, allowing you to incorporate knowledge, but you must be transparent about it.

**Bayesian inference gives a single answer** — After updating, I get one number that is the true probability of the event.

The posterior is a distribution, not a single point. It represents your belief, including its uncertainty. You can summarise it with a point estimate, but that loses information. In fact: The posterior is a full distribution. Use it to compute credible intervals and to make decisions that account for uncertainty.


## In short

Bayesian inference is about updating beliefs with evidence using Bayes' rule: posterior ∝ prior × likelihood. Decision theory then takes the posterior and chooses the action that minimises expected loss. The posterior carries uncertainty; the loss function carries what you care about. Optimal decisions require both. If you remember nothing else: the posterior is the belief, the loss is the preference, and the best action balances them.


## Where this is used

Bayesian inference and decision theory are not just theoretical — they are used in real AI systems in several critical ways.\n\n**Large Language Models (LLMs):** LLMs like ChatGPT generate text by predicting the next word. The model assigns probabilities to all possible next words (a probability distribution, essentially a posterior over the next token). Decision theory comes in when choosing which word to output. A simple decision rule is to pick the most probable word (greedy decoding). But that can lead to repetitive text. Instead, LLMs use sampling strategies that balance the expected quality (a kind of loss) against diversity. The temperature parameter, for example, adjusts how evenly the probabilities are spread — a high temperature makes the distribution flatter, exploring more possibilities, while a low temperature makes it more peaked, sticking to safe choices. This is a decision under uncertainty, optimising a trade-off between accuracy and creativity.\n\n**Small Models on Devices (e.g., keyword spotting):** On a smartphone or smart speaker, a small model listens for a wake word like 'Hey Siri.' The model must decide quickly and with limited compute whether the audio contains the wake word. It computes a probability that the wake word was spoken (a posterior). Then it applies a decision rule: if the probability exceeds a threshold, it activates. The threshold is a decision-theoretic trade-off between false positives (waking up when you didn't say the wake word) and false negatives (missing the wake word). Setting the threshold too low causes frequent false alarms; too high makes the device unresponsive. Decision theory helps choose the threshold that minimises the expected number of errors, accounting for the costs of each.\n\n**Retrieval-Augmented Generation (RAG):** In RAG, a system retrieves relevant documents from a database and feeds them to an LLM to answer a question. Bayesian inference can be used to rank documents. Each document has a relevance score, which can be interpreted as a posterior probability that the document is relevant given the query. The system might also model the uncertainty in the answer: if the retrieved documents are contradictory, the posterior distribution over possible answers is flat, and the system might decide to ask a clarifying question rather than give a single answer. This is a decision under uncertainty about which action (give answer or ask) maximises user satisfaction.\n\n**Agentic Systems (planning and tool use):** An AI agent that plans and calls tools (like a web search or a calculator) must decide what to do next. It has a belief state (a posterior over the environment). It uses decision theory to choose actions that maximise expected reward (or minimise expected loss). For example, if the agent must book a flight, it might believe there is a 70% chance the price will drop tomorrow. It then compares the expected cost of booking now vs. waiting. It might also decide to gather more information (e.g., check historical price data) before acting, which is a form of decision-making under uncertainty. The agent weighs the cost of waiting (price might rise) against the benefit (price might fall).\n\n**Everyday Applied ML (fraud detection, recommendation systems, forecasting):** Fraud detection is a classic decision problem. A model outputs a probability that a transaction is fraudulent. The company then decides whether to block the transaction, require additional verification, or allow it. The loss function includes the cost of a false block (losing a legitimate customer) versus a false allow (losing money to fraud). Decision theory sets the optimal threshold.\n\nRecommendation systems: When a streaming service recommends a movie, it predicts the probability you'll like it (posterior). The decision could be to show the movie or not, but also to decide which movie to show when there is competition for the top slot. The loss might be the expected viewing time or the chance of you disengaging. The system chooses the recommendation with the highest expected benefit, factoring in uncertainty.\n\nForecasting: Weather prediction models output probability distributions over future temperatures or rainfall. Decision theory then helps utilities, farmers, and cities decide whether to take action (e.g., irrigate, issue warnings) based on the expected loss of being wrong.\n\n**Speech recognition:** In speech-to-text, the system produces a distribution over possible words or phrases given the audio. Decision theory selects the most likely sequence (or the one with minimal expected error, often measured by word error rate). Sometimes it uses a language model to bias the probabilities, which is a form of prior.\n\n**In practice, a developer writing code for these systems uses libraries like PyTorch or TensorFlow for the neural networks, and then implements decision rules in Python. For example, in a fraud detection system, a developer might write:**\n\n```python\nimport torch\nmodel = torch.nn.Sequential(...)  # neural network\nprob = torch.sigmoid(model(features))  # posterior probability\nif prob > 0.3:  # threshold from decision theory\n    flag_for_review()\nelse:\n    allow_transaction()\n```\n\nThe threshold is chosen based on the business's cost of false positives and false negatives. In more sophisticated systems, Bayesian optimisation is used to tune hyperparameters, which itself uses a posterior over model performance to decide which hyperparameter to try next.\n\n**Breakdown when done badly:** If the posterior is miscalibrated (overconfident or underconfident), the decisions will be poor despite good expected loss calculations. A common mistake is to ignore the prior, leading to overfitting or biased decisions. Another is to use the wrong loss function — e.g., using accuracy instead of a cost-sensitive loss can lead to suboptimal decisions in imbalanced settings like rare disease detection.


---

Read this on the web: https://basavapurushottam.com/learn-ai/bayesian-inference-and-decision-theory
