# Inverse Problems

Recovering unknown causes from observed effects, typically ill-posed because many causes explain the same observations.

**Rests on:** Convex Optimization, Gradient Descent

*Mathematical Foundations · Optimization*


## Why learn this

Every AI system is solving an inverse problem, and recognising that is the difference between understanding what a model does and just running it. In image denoising, you observe a noisy photo and infer the clean original. In super-resolution, you observe a small blurry image and infer the sharp large one. In medical imaging, you observe X-ray measurements and infer the internal structure. Even training a neural network is an inverse problem: you observe training examples and infer the weights that produced them. This concept unlocks the tool that makes all such problems tractable — regularisation (a technique for adding extra constraints to narrow down the possible answers) — a principle you will see repeatedly in later work on finding optimal solutions and understanding how complex models behave. Master this page and you will never again wonder why different models fit the same data in different ways.


## The intuition

Start with something you do every day: you hear a thud outside and think ‘that was a coconut falling from the tree.’ You did not see the coconut. You heard a sound, and from that sound you guessed the cause. That is an inverse problem: from an observed effect (the sound) you recover an unknown cause (the coconut). Now imagine hearing the same thud but realising it could have been a coconut, a large mango, or a neighbour dropping a bag of rice. All three causes produce exactly the same sound. This is the core difficulty of inverse problems: many different causes can lead to the same observation. The sound alone does not tell you which one it was. The same thing happens in AI. A blurry photo could come from a moving camera, a dirty lens, or a low-resolution sensor. A low sales figure could come from high prices, bad marketing, or a competitor’s launch. The observations alone do not tell you the cause. Inverse problems are everywhere, and the hard part is that they are almost always ambiguous. This page teaches you why that ambiguity exists and what mathematicians do about it.


## An analogy

Imagine you are in a dark room and you hear a knock on the wall. You only hear one knock, and you want to know what made it. It could be a person knocking, a falling branch, or a rat moving. All three create the same sound. You cannot see the cause, only the effect. In mathematics, the effect is the data you measure (the knock), and the cause is what you want to find (the object). An inverse problem is exactly this: given the effect, find the cause. The analogy breaks down when you realise that in a real room, you might gather more information — knock again, listen from different positions, or shine a light. In AI, you often cannot get more data; you only have the one observation, and you must make your best guess. This is why inverse problems are ill-posed: the data alone does not pin down a single answer. The act of choosing one answer among many is what regularisation does — it adds a rule for which cause is more likely. Without that rule, you are stuck guessing.


## Definition

An inverse problem is the task of recovering unknown causes from observed effects, where the mapping from causes to effects is not one-to-one, so many different causes can explain the same observations.


## How it connects

This is your first concept in the Optimization family. The library notes say inverse problems belong to Optimization, and you will see why: solving an inverse problem is an optimisation task — you search over possible causes to find the one that best explains the data. You have not yet studied Convex Optimization, which is a sibling under the same parent; that concept helps when the space of possible solutions has a simple, bowl-shaped structure where any path downhill leads to the same bottom point. Inverse problems are often not convex, which is why they are hard. The foundational idea here — that many causes fit the same observations — is also the seed of Double Descent, where a model with too much capacity fits the data in many ways and handles that ambiguity differently. For now, focus on the core: the inverse mapping is ambiguous, and the remedy is a rule that prefers some causes over others.


## The mathematics

An inverse problem asks you to work backwards from what you see to what caused it. Imagine you have a machine that takes some hidden causes and turns them into visible observations. You know how the machine works, but you only get to see the output. Your job is to guess the input.

To do this, we use lists of numbers called vectors and tables of numbers called matrices. A vector is just a list. For example, let x be a list containing two unknown causes: x1 and x2. Let b be a list containing two observed values: b1 and b2. The machine’s behavior is described by a matrix A. This matrix tells you how much each cause contributes to each observation.

The relationship between the causes and the observations is written as:
b = A * x.

Here, "A * x" means we multiply the table A by the list x. Specifically, the first number in b (b1) is calculated by taking the first row of A, multiplying each entry by the corresponding cause in x, and adding them up. The second number in b (b2) is calculated similarly using the second row.

For example, suppose the matrix A is:
[ 2  1 ]
[ 3  4 ]

And the causes are x1 = 10 and x2 = 5. Then the observations would be:
b1 = (2 * 10) + (1 * 5) = 20 + 5 = 25.
b2 = (3 * 10) + (4 * 5) = 30 + 20 = 50.

So the observed list b is [25, 50].

In an inverse problem, you are given A and b, and you need to find x. If the matrix A is "nice" (mathematically, if it has an inverse), you can just reverse the operation: x = A-inverse * b. But often, A is not nice. It might be rectangular (more observations than causes, or vice versa) or singular (the rows are redundant). In these cases, there is no exact solution that makes b exactly equal to A * x.

Instead, we look for the best guess. We want to find an x that makes the predicted output (A * x) as close as possible to the observed output (b). "Close" is measured by the sum of squared differences. This is called the least squares method.

We calculate the difference between each observed value and its predicted value, square that difference, and add them all up. We want to make this total sum as small as possible.

Let the predicted values be p1 and p2. The error E is:
E = (b1 - p1)^2 + (b2 - p2)^2.

Since p1 and p2 depend on x, we write this as minimizing E over all possible x.

However, even with least squares, there can be many different x values that give the same minimum error. This is called ill-posedness. To pick one specific answer, we add extra information. We assume the causes should have certain properties, such as being small or smooth. This assumption is called a prior.

We add a penalty term to our error function. This penalty discourages solutions that violate our prior belief. For example, if we believe the causes should be small, we add the sum of the squares of the causes, multiplied by a weight lambda (λ).

The new function to minimize is:
Total Error = (b1 - p1)^2 + (b2 - p2)^2 + λ * ((x1^2) + (x2^2)).

Here, λ is a positive number that controls the trade-off. A large λ means we care more about keeping the causes small, even if it means the fit to the data is worse. A small λ means we care more about fitting the data exactly.

This combined approach is called Tikhonov regularization. It turns an impossible or ambiguous inverse problem into a well-defined optimization problem: find the x that minimizes the Total Error.

Symbol glossary:
x = the list of unknown causes (vector).
b = the list of observed values (vector).
A = the matrix describing how causes produce observations.
p = the predicted values (A * x).
λ = the regularization strength (a positive number).
R(x) = the regularizer, here defined as the sum of squares of x.


## A worked example

You are a doctor trying to determine the size of a tumour from two X-ray measurements. Let the unknown size be a number T (in cm). The X-ray machine is imperfect: the first measurement is affected by T and by the machine noise. To keep it simple, suppose the first measurement M1 is 1.5 times the tumour size plus a small constant 0.2. The second measurement M2 is 0.8 times the tumour size plus a constant 0.1. You observe M1 = 2.6 and M2 = 1.4. You want to find T.

Step 1: Write down the equations for the observations.
M1 = 1.5 * T + 0.2
M2 = 0.8 * T + 0.1

Step 2: Substitute the observed values into the equations.
2.6 = 1.5 * T + 0.2
1.4 = 0.8 * T + 0.1

Step 3: Solve the first equation for T.
Subtract 0.2 from both sides:
2.6 - 0.2 = 1.5 * T
2.4 = 1.5 * T
Divide by 1.5:
T = 2.4 / 1.5
T = 1.6

Step 4: Check if this value of T works for the second equation.
Calculate the predicted M2 using T = 1.6:
Predicted M2 = 0.8 * 1.6 + 0.1
Multiply 0.8 by 1.6:
0.8 * 1.6 = 1.28
Add 0.1:
1.28 + 0.1 = 1.38

Step 5: Compare the predicted M2 with the observed M2.
Observed M2 = 1.4
Predicted M2 = 1.38
Difference = 1.4 - 1.38 = 0.02

The difference is not zero. This means there is no single value of T that satisfies both equations exactly. The system is inconsistent due to noise or measurement error.

Step 6: Use the least squares method to find the best T.
We want to minimize the sum of squared errors:
Error = (Observed M1 - Predicted M1)^2 + (Observed M2 - Predicted M2)^2

Let's express the predicted values in terms of T:
Predicted M1 = 1.5 * T + 0.2
Predicted M2 = 0.8 * T + 0.1

Substitute these into the error formula:
Error(T) = (2.6 - (1.5 * T + 0.2))^2 + (1.4 - (0.8 * T + 0.1))^2

Simplify the terms inside the parentheses:
Term 1: 2.6 - 0.2 - 1.5 * T = 2.4 - 1.5 * T
Term 2: 1.4 - 0.1 - 0.8 * T = 1.3 - 0.8 * T

So:
Error(T) = (2.4 - 1.5 * T)^2 + (1.3 - 0.8 * T)^2

To find the minimum, we would normally take the derivative with respect to T and set it to zero. But since you haven't met derivatives, let's look at the structure. We are minimizing a sum of two squared linear functions. This is a convex parabola opening upwards. The minimum occurs where the slope is zero.

Alternatively, we can use the normal equation for least squares. For a single variable T, the best fit minimizes the error when:
Sum of (Residual * Derivative of Prediction) = 0

The residual for M1 is (2.6 - (1.5T + 0.2)). The derivative of the prediction (1.5T + 0.2) with respect to T is 1.5.
The residual for M2 is (1.4 - (0.8T + 0.1)). The derivative of the prediction (0.8T + 0.1) with respect to T is 0.8.

So we set:
(2.6 - 1.5T - 0.2) * 1.5 + (1.4 - 0.8T - 0.1) * 0.8 = 0

Simplify:
(2.4 - 1.5T) * 1.5 + (1.3 - 0.8T) * 0.8 = 0

Expand:
(2.4 * 1.5) - (1.5T * 1.5) + (1.3 * 0.8) - (0.8T * 0.8) = 0

Calculate the products:
2.4 * 1.5 = 3.6
1.5 * 1.5 = 2.25
1.3 * 0.8 = 1.04
0.8 * 0.8 = 0.64

Substitute back:
3.6 - 2.25T + 1.04 - 0.64T = 0

Combine constant terms:
3.6 + 1.04 = 4.64

Combine T terms:
-2.25T - 0.64T = -2.89T

So:
4.64 - 2.89T = 0

Solve for T:
2.89T = 4.64
T = 4.64 / 2.89
T ≈ 1.6055

Result: The best estimate for the tumour size is approximately 1.61 cm. This value minimizes the total squared error across both measurements, providing a single answer despite the inconsistency in the raw data.


## From scratch, in Python

```python
# We will solve a simple inverse problem: recover the cause x from observations b
# using least squares and regularisation.

# First, we need to learn about lists and loops.
# A list is an ordered collection that can be changed. For example:
prices = [10, 20, 30]
quantities = [1, 2, 3]

# To compute the dot product, we loop over the lists and multiply corresponding entries.
def dot(a, b):
    total = 0
    for i in range(len(a)):
        total = total + a[i] * b[i]
    return total

# Test the dot function.
print(dot(prices, quantities))  # 10*1 + 20*2 + 30*3 = 80

# Now, for the inverse problem, we set up a simple case: two observations, one cause.
# Equations: b1 = 2*x, b2 = 3*x
# We observe b1 = 4.8, b2 = 7.2. What is x?
# Best fit is the x that minimises (2x - 4.8)^2 + (3x - 7.2)^2
# We can try a few values of x and compute the cost.

def cost(x):
    b1_pred = 2 * x
    b2_pred = 3 * x
    return (b1_pred - 4.8) ** 2 + (b2_pred - 7.2) ** 2

# Search over a grid of x values.
best_x = None
best_cost = float('inf')
for x in range(0, 100):  # x from 0 to 99
    x_val = x / 10  # so x from 0.0 to 9.9
    c = cost(x_val)
    if c < best_cost:
        best_cost = c
        best_x = x_val

print(f"Best x = {best_x}, cost = {best_cost}")
# The exact solution is x = 2.4, but the grid finds a close value.

# Now add regularisation. We penalise large x by adding lambda * x^2.
lambda_val = 0.1
def regularised_cost(x):
    return cost(x) + lambda_val * x * x

# Find the regularised best x.
best_reg_x = None
best_reg_cost = float('inf')
for x in range(0, 100):
    x_val = x / 10
    c = regularised_cost(x_val)
    if c < best_reg_cost:
        best_reg_cost = c
        best_reg_x = x_val

print(f"Regularised best x = {best_reg_x}, cost = {best_reg_cost}")
# Regularisation pulls x slightly toward zero.
```


## The same thing, with a framework

```python
import numpy as np

# Use numpy to solve a least squares problem.
# A is the matrix of coefficients, b is the observed data.
# In this example, A is a 2x1 matrix, b is a vector of two observations.
A = np.array([[2.0], [3.0]])
b = np.array([4.8, 7.2])

# The least squares solution is x = (A^T A)^{-1} A^T b
# numpy provides lstsq which does exactly that.
x_hat, residuals, rank, s = np.linalg.lstsq(A, b, rcond=None)
print(x_hat)  # approximately [2.4]

# With regularisation, add lambda * I to A^T A.
lambda_val = 0.1
A_T_A = A.T @ A + lambda_val * np.eye(1)  # np.eye(1) is the 1x1 identity matrix
A_T_b = A.T @ b
x_reg = np.linalg.solve(A_T_A, A_T_b)
print(x_reg)  # slightly less than 2.4
```


## What the code does

The from-scratch code introduces two fundamental Python concepts. A list is a collection of values that can be changed; we create two lists of numbers and multiply them together using a loop. The loop is a for statement that repeats the same operation over each index. We define a function called dot that computes the dot product of two lists — it sums the products of corresponding entries, which is the same operation you performed by hand in the mathematics section. Next, we set up a simple inverse problem where the relationship is linear: two equations predicting two observations from one unknown x. We define a cost function that measures how far the predicted observations are from the actual ones, using squared differences. We then search over possible x values from 0.0 to 9.9 in steps of 0.1, compute the cost for each, and keep the smallest. This is a brute-force optimisation. Finally, we add regularisation by adding lambda_val * x * x to the cost — this penalises large x and shifts the best answer slightly. The framework version uses NumPy, which is a library that provides efficient operations on arrays. The last line uses np.linalg.lstsq, which solves the least squares problem directly without a search. Under the hood, it uses a matrix operation that corresponds to the normal equations, A^T A x = A^T b. The regularised version adds lambda times the identity matrix to A^T A before solving, which is exactly what the from-scratch code did by adding the penalty to the cost.


## Common misconceptions

**More data always gives a unique answer** — If I collect enough observations, I will definitely be able to find the true cause.

Ill-posedness persists even with infinite data if the cause is not identifiable from the observations. For example, in a mathematical sense, if the matrix A is singular, no amount of b will pin down a unique x. In fact: The structure of the mapping between cause and effect determines whether the answer is unique, not just the amount of data.

**The solution that minimises the mismatch is correct** — The cause that best explains the data is the true cause.

Many different causes can explain the same data equally well; the best-fitting one may still be wrong. In fact: The data alone does not tell you the truth; you need to add a prior or regularizer to choose among equally good fits.

**Regularisation is a fix for overfitting only** — Regularisation is just a trick to avoid overfitting a model.

Regularisation is fundamental to making inverse problems well-posed. Without it, many inverse problems have no stable solution at all. In fact: Regularisation adds stability and uniqueness to ill-posed inverse problems; it also helps with overfitting, but that is a secondary benefit.

**If the model fits the data well, it is good** — Low training error means the model is accurate.

Because of ill-posedness, you can find a model that fits the training data perfectly but performs poorly on new data. In fact: Good performance on new data requires that the chosen solution generalises, which often means sacrificing perfect fit on training data.

**Inverse problems are only in image processing** — Inverse problems are about images, not about AI in general.

Any task that infers hidden causes from observed data is an inverse problem, spanning text, audio, and tabular data. In fact: Inverse problems are a general framework that appears in every domain where you infer from observations.

**An ill-posed problem is a badly designed problem** — Ill-posed problems are just errors in the data or the model.

Ill-posedness is a structural property of many real-world problems, not a mistake. In fact: Many real problems are inherently ill-posed; regularisation is the standard way to make them solvable.


## In short

Inverse problems recover causes from effects, but the mapping is often not one-to-one — many causes explain the same observations. This ill-posedness is fundamental, not a bug. The remedy is regularisation: adding a prior that selects a plausible solution. The mathematics involves linear algebra (matrices and dot products) and optimisation (minimising the mismatch). In AI, every model training is an inverse problem, and regularisation is what makes models generalise instead of memorise.


## Where this is used

Inverse problems are everywhere in AI, and the tools you have learned here are what make every one of them work. Start with large language models. The training of an LLM is an inverse problem: you have the text (the observations) and you want to find the weights of the neural network (the causes) that explain the text. The training data are the observed effects, and the weights are the unknown causes. The model’s prediction is the forward mapping; the training process is the inverse. Regularisation is built in through weight decay, which adds a penalty on the size of the weights, exactly the λ·R(x) term you met here. Without it, the model would overfit — it would find weights that memorise the training text but do not generalise. Every LLM trainer sets a weight decay value, and that is a regularisation strength. When you hear of a model ‘hallucinating’, one cause is that the inverse problem is ill-posed: many weight configurations fit the training data, and some of those produce false but confident outputs. Now move to small language models that run on a phone. These are designed with regularisation in mind from the start: they use fewer parameters, which acts as a form of regularisation by restricting the set of possible solutions. When you use a tiny model for autocomplete on your keyboard, the model is solving an inverse problem: given the last few words (the observation), infer the next word (the cause). There are many possible next words, making the problem ill-posed. The model uses a prior from its training data — the distribution of words in the language — to choose the most likely one. In retrieval-augmented generation, you have a query and you want to retrieve a relevant document. This is an inverse problem: the query is an effect, the document is the cause. Embeddings are the tool that lets you compare causes (documents) and effects (queries) in a shared space. The embedding of a query is a point in a high-dimensional space, and the retrieval system looks for documents whose embeddings are close to the query’s embedding. This closeness is exactly the dot product you computed in the mathematics section. Ill-posedness arises because many documents look similar for a given query; regularisation comes in through the choice of embedding model, which has its own regularisation to make embeddings distinctive. Agentic systems that plan and call tools face an inverse problem at every step: given the goal and the current state, what is the best action to take? There are many actions that could explain the same state, and the system uses a policy learned from experience, which encodes a prior over which actions are plausible. The planning loop itself is an optimisation: search over possible sequences of actions (causes) to find the one that leads to the desired outcome (the observed effect). In everyday applied machine learning, inverse problems are literally everywhere: image denoising (infer clean image from noisy observation), super-resolution (infer high-res from low-res), medical imaging (infer tissue from X-ray projections), weather forecasting (infer current state from past observations), and fraud detection (infer fraudulent activity from transaction patterns). In each case, the practitioner writes code that sets up the forward model, defines a loss function measuring mismatch, and adds a regulariser. For example, in PyTorch, a developer creating a neural network for image denoising would define a model that maps a noisy image to a clean one, train it with a loss that combines the squared difference between the predicted and true clean image (the data term) and a penalty that encourages smoothness in the output (the regulariser). The developer controls the regularisation strength with a hyperparameter, and tuning that hyperparameter is often the difference between a model that produces blurry but stable images and one that produces sharp but noisy ones. What breaks when this is done badly is exactly what you learned: without regularisation, the model produces wildly different answers for slightly different noisy inputs — the ill-posedness that this page describes. With too much regularisation, the model produces overly smooth, unhelpful results, ignoring the actual data. The skill of an AI engineer is choosing the right regulariser and the right strength for the problem.


---

Read this on the web: https://basavapurushottam.com/learn-ai/inverse-problems
