# ELBO

The evidence lower bound, the tractable objective variational inference maximises in place of an intractable marginal likelihood.

**Rests on:** Bayesian Inference and Decision Theory, Shannon Entropy

*Mathematical Foundations · Information Theory*


## Why learn this

The ELBO is the number that allows computers to learn how to create new images, text, or audio from scratch. When you use a system that generates a face that never existed, or writes a story in your style, it is using this concept. The problem these systems face is mathematical: they need to work out which hidden features (like 'eye color' or 'grammar rule') produced the output, but calculating the exact probability of every possible combination is too slow and complex for any computer to handle directly. The ELBO solves this by providing a reliable, calculable number that tells the system exactly how well it is doing. By maximizing this number, you teach the system to generate realistic data while understanding the hidden structures that created it. Without this trick, modern artificial intelligence would be unable to learn from experience in complex ways.


## The intuition

Imagine you are looking at a gallery of handwritten digits and trying to guess the 'story' behind each one. You know that every digit is caused by two things: which number it is (0 through 9), and how it was written (the angle, the slant, the thickness). However, you cannot see those causes directly; you only see the final image. To understand the data, a computer would ideally need to calculate the probability of every possible combination of cause and effect. But there are infinitely many ways a digit could be tilted or thickened, making the exact calculation impossible. The ELBO saves the day by taking a shortcut. Instead of finding the truth exactly, you guess a simpler pattern that roughly matches reality—for example, assuming all digits are upright and perfectly round. You then calculate a number called the ELBO based on this simple guess. This number has two parts: first, how well your simple guess explains the data (did you draw a digit that looks like the real one?); second, how far off your simple guess is from the complex reality (did you ignore too much detail?). The goal is to maximize this ELBO number. A higher ELBO means your simplified model is both accurate enough to recreate the data well and simple enough not to be confused by noise. It allows the computer to iteratively refine its 'simplified' understanding until it can generate new, plausible examples that fit the original patterns.


## An analogy

Think of trying to describe a complex movie plot to a friend using only a single word. You want to pick a word that captures the essence of the story. The ELBO is like a score given to your choice of that one word. The score looks at two things: does the word help your friend imagine the correct scenes (the reconstruction quality), and is the word too vague or misleading (the simplicity penalty)? You keep changing your chosen word to raise the score, finding the sweet spot where the word is informative but not overly complicated. A minor caveat: unlike a single word for one movie, the ELBO evaluates this trade-off across thousands of different movies simultaneously.


## Definition

The ELBO is a calculable value that provides a safe lower limit for how well a model explains observed data, and it serves as the primary objective function that machine learning algorithms adjust to improve their ability to generate new samples by approximating complex hidden structures.


## How it connects

This concept sits alongside Variational Inference, which is the broader method of using approximate distributions to solve difficult probability problems, and Variational Autoencoders, which are specific neural network architectures that use the ELBO as their training rule to learn compressed representations of data.


## The mathematics

The ELBO is a quantity from a branch of mathematics called information theory. Information theory is about measuring uncertainty: how surprised you are by an outcome, how many bits it takes to encode a message. The ELBO is built from two ideas. The first is how well your model explains the data: given a guessed cause, how likely is the data? That is the reconstruction term — it rewards a model that can recreate the observed data from the guessed cause. The second idea is how far your guess is from a sensible one: KL divergence measures the extra cost of using one distribution when another is correct. You want your guess to be simple and not too far from the true cause. The ELBO is simply the reconstruction term minus that KL divergence. You maximise it: you adjust your model to increase the first term and decrease the second. That single number, when maximised, simultaneously makes the model better at explaining the data and keeps the guess plausible. The name 'ELBO' stands for Evidence Lower Bound: the log evidence is the quantity you truly want but cannot compute, and the ELBO is a lower bound on it that you can compute and maximise, and by doing so you also push that bound up towards the true evidence.

ELBO = E_{z~q}[log p(x|z)] - KL(q(z) || p(z))

Where:
- q(z) is your chosen approximate posterior distribution over latent causes.
- p(x|z) is the likelihood: how probable is the observed data x given a cause z.
- p(z) is the prior distribution over causes.
- E_{z~q}[...] is the average of the log-likelihood over samples from q.
- KL is the Kullback-Leibler divergence between q and the prior.


## A worked example

```
Suppose you have one data point: a simple image that is just a single value, say the brightness 5. You have a model that, given a cause (a latent variable), predicts the brightness. Your cause is a single number z. Your model says: the brightness is roughly z plus some noise. You also have a prior distribution over z: it should be around 0 with some spread. You choose an approximate posterior q(z) that is also centered near the observed data.

Step 1: Reconstruction: observed brightness = 5, predicted from q: mean of z = 4.2   # We sample a z from q, say z=4.2, and the model predicts the brightness as z (for simplicity). The reconstruction term is the log probability of observing 5 given this predicted mean, assuming Gaussian noise with variance 1. Log probability ≈ -((5-4.2)^2)/2 = -0.32.
Step 2: KL term: q is N(4.2, 1), prior is N(0, 1). KL(q||prior) = ( (4.2-0)^2 + 1 - 1 )/2 = 8.82   # Here we compute how far q is from the prior. Since both variances are 1, the KL simplifies to half the squared difference of the means.
Step 3: ELBO = reconstruction - KL = -0.32 - 8.82 = -9.14   # We subtract the KL from the reconstruction. This is a negative number because the KL is large.
Step 4: Now let q be centered at 4.8. Reconstruction = -(5-4.8)^2/2 = -0.02, KL = ((4.8-0)^2)/2 = 11.52   # Moving q closer to the data improves reconstruction but increases the KL penalty because q moves further from the prior (0).
Step 5: New ELBO = -0.02 - 11.52 = -11.54   # This is worse! The KL penalty grew faster than the reconstruction improved.
Step 6: What if q is centered at 4.0? Reconstruction = -(5-4)^2/2 = -0.5, KL = (4^2)/2 = 8.0, ELBO = -0.5 - 8.0 = -8.5   # Here the balance is better: a slightly worse reconstruction but a much smaller KL term yields a higher ELBO.

Result: The best ELBO among the candidates we tried is -8.5, achieved when q is centered at 4.0. This shows that the optimal q is not exactly at the data point but is pulled toward the prior — the ELBO balances reconstruction fidelity against simplicity. In practice, you'd search over all possible q parameters to find the maximum.
```


## From scratch, in Python

```python
import numpy as np

def elbo(x, mu_q, sigma_q, mu_prior=0.0, sigma_prior=1.0):
    """Compute the ELBO for a simple Gaussian model with a scalar latent."""
    # reconstruction term: average log-likelihood over samples from q
    samples = np.random.normal(mu_q, sigma_q, size=1000)
    # assume p(x|z) is Gaussian with mean z and variance 1
    log_likelihoods = -0.5 * ((x - samples) ** 2) - 0.5 * np.log(2 * np.pi)
    reconstruction = np.mean(log_likelihoods)

    # KL(q||prior): both are Gaussians
    kl = np.log(sigma_prior / sigma_q) + (sigma_q**2 + (mu_q - mu_prior)**2) / (2 * sigma_prior**2) - 0.5

    return reconstruction - kl

# test with a single data point and some parameter choices
x = 5.0
for mu in [4.0, 4.2, 4.8]:
    print(f"mu={mu}, ELBO={elbo(x, mu, 1.0):.3f}")
```


## The same thing, with a framework

In PyTorch, a variational autoencoder (VAE) uses the ELBO as its loss. The encoder outputs parameters for `q`, and the decoder predicts the reconstruction. The loss is:

```python
import torch

def vae_loss(recon_x, x, mu, logvar):
    # reconstruction term: binary cross-entropy or MSE between x and recon_x
    BCE = torch.nn.functional.binary_cross_entropy(recon_x, x, reduction='sum')
    # KL divergence between q(z) = N(mu, exp(logvar)) and standard normal
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return BCE + KLD
```

This is exactly the same as our plain Python: the reconstruction term is the average log-likelihood (here, cross-entropy), and the KL term is the same formula, just computed in a batch and in log-variance form for numerical stability.


## What the code does

We implement the ELBO in plain Python. First we import numpy for random numbers and basic math. We define a function `elbo` that takes an observed data point `x`, and parameters of our approximate posterior: `mu_q` (mean) and `sigma_q` (standard deviation). Inside, we sample 1000 values from `q` using `np.random.normal`. For each sample, we compute the log-likelihood of `x` given that sample as the mean of `x` — we assume the likelihood is a Gaussian centred at `z` with variance 1, so `log p(x|z) = -0.5*(x-z)^2 - 0.5*log(2*pi)`. We then average these to get the reconstruction term. Next we compute the KL divergence between `q` and a standard normal prior (mean 0, variance 1) using the known formula for two Gaussians. Finally we return the difference. The loop tests a few means, showing how the ELBO changes — you'll see that 4.0 gives a higher ELBO than 4.2 or 4.8.


## Common misconceptions

**ELBO is the log-evidence itself** — The ELBO is just the log-probability of the data.

It is a lower bound, not the true value — it is always less than or equal to the log-evidence. In fact: The ELBO is a bound that is maximised to approach the true log-evidence, but they are not the same.

**Maximising ELBO only improves reconstruction** — If I maximise the ELBO, I only need to make the model reconstruct the data well.

The KL term prevents simply memorising the data; it also enforces a sensible latent distribution. In fact: Maximising the ELBO balances reconstruction quality with the simplicity of the latent distribution.

**KL term is always positive, so ELBO is always negative** — Since KL is positive, the ELBO is always negative, so maximizing it means making the least negative number.

The reconstruction term can be positive (log-likelihood can be positive), and the bound can be positive for well-fitting models. In fact: ELBO can be positive; the reconstruction term can outweigh the KL divergence.

**ELBO is only for VAEs** — The ELBO is just a fancy name for the VAE loss.

Variational inference is a general technique used in many probabilistic models, and the ELBO is its core objective. In fact: The ELBO is the objective for variational inference in any latent-variable model, VAEs being one special case.

**A higher ELBO always means a better model** — If my ELBO is higher, my model is more accurate.

The ELBO is a lower bound; a higher bound might indicate a better fit, but it can also indicate overfitting to the training data. In fact: A higher ELBO on training data generally indicates a better model fit, but you must check generalisation to unseen data.


## In short

The ELBO is a tractable lower bound on the log-evidence, decomposing into a reconstruction term and a KL regulariser. Maximising it approximates maximum-likelihood learning when the full likelihood is intractable, and it is the loss function used to train variational autoencoders. Remember: the bound is tight when the approximate posterior matches the true posterior — then the ELBO equals the log-evidence.


## Where this is used

The ELBO is the objective that trains variational autoencoders (VAEs), which are the backbone of many generative models. In standard VAEs, the encoder network maps data to a distribution over latent variables, and the decoder reconstructs the data. The ELBO loss is used to update both networks simultaneously. In large language models (LLMs), the ELBO appears as the evidence lower bound when working with probabilistic latent variable models, though LLMs themselves are trained with cross-entropy loss. However, for models like DALL·E and other text-to-image systems, VAEs are used to compress images into a latent space, and the ELBO is what trains those VAEs. For small models running on devices, such as on a phone, VAEs trained with the ELBO are used for anomaly detection (e.g., recognising unusual sensor readings) and for on-device image compression. In retrieval-augmented generation (RAG), the embedding models that map text and queries to vectors are not typically trained with the ELBO, but VAEs have been used for document embeddings; the principle of having a tractable objective between two distributions applies. In agentic systems that plan and call tools, the ELBO is not usually the objective, but variational inference techniques are used in Bayesian optimisation and reinforcement learning (e.g., variational policy gradients). The ELBO is most relevant in everyday applied machine learning: building a VAE for anomaly detection, using it to generate synthetic data, or for dimensionality reduction. A developer would write something like the PyTorch code above, train the model with a gradient-based optimiser, and monitor the ELBO (or its negative as loss) during training. When done badly, maximising the ELBO can lead to posterior collapse, where the latent variable is ignored, and the model generates blurry or low-quality samples. This happens when the KL term dominates and the reconstruction is poor, so you need to balance the two.


---

Read this on the web: https://basavapurushottam.com/learn-ai/elbo
