# Least Squares Regression

Fitting parameters by minimising the sum of squared residuals, the oldest and still most-used estimation criterion.

**Rests on:** Normal Equations

**Leads to:** Dimensionality Reduction and PCA, Linear Regression, Loss Function Selection, Polynomial Regression, Ridge and Lasso Regression

*Mathematical Foundations · Optimization*


## Why learn this

You often need to find a single rule that describes a messy collection of numbers. Imagine you have a list of sales figures for different months and want to predict next month's sales. Or perhaps you are comparing the price of used cars against their age, looking for the general trend. Least squares regression is the standard tool for this. It finds the straight line (or curve) that sits closest to your data points, turning a cloud of scattered information into a clear, usable pattern. You use it whenever you want to say, 'Given these inputs, what is the most likely output?' without guessing wildly.


## The intuition

Let's look at a concrete example. Suppose you have three data points representing house prices (in tens of thousands of dollars) based on their size (in hundreds of square feet):
1. Size 20, Price $100
2. Size 40, Price $150
3. Size 60, Price $200

You want to draw a line through these. The line might not hit every point exactly. For each house, you measure the vertical gap between the actual price and the line's predicted price. This gap is called the error.

For point 1, if the line predicts $95, the error is 5 (100 - 95).
For point 2, if the line predicts $160, the error is -10 (150 - 160).
For point 3, if the line predicts $205, the error is -5 (200 - 205).

If you just added these errors (5 + (-10) + (-5)), you get 0. This is misleading because the line is clearly not a perfect fit; it misses points by significant amounts. To fix this, we square each error. Squaring makes every gap positive and exaggerates larger gaps.

Squared errors:
1. 5^2 = 25
2. (-10)^2 = 100
3. (-5)^2 = 25

Total squared error = 150.

The 'Least Squares' method searches for the specific line that makes this total sum as small as possible. Why square instead of just taking absolute values? Because squaring creates a smooth, bowl-shaped surface of errors. This shape has one clear bottom point, allowing us to calculate the exact best line using algebra, rather than having to guess and check endlessly.


## An analogy

Imagine you are holding a rubber band attached to a nail in a board. The nail represents your data point. You pull the rubber band taut so that it tries to sit as close to all nails as possible simultaneously. The tension in the rubber band represents the error. When you let go, the rubber band snaps into the position where the total tension (the sum of squared stretches) is minimized. This visualizes how the line settles into the 'valley' of lowest total error. Note that a single nail pulled far away (an outlier) will stretch the rubber band immensely, pulling the line toward it disproportionately strong.


## Definition

Least squares regression is a technique for finding the best-fitting line or curve by choosing its settings—such as steepness and starting point—to minimize the sum of the squared vertical distances between each data point and the line.

In plain terms: It picks the equation that reduces the total amount of 'squaring-and-adding-up' error to its lowest possible value.


## How it connects

This concept is the bedrock for two adjacent ideas. First, **gradient descent** is a general method for finding the bottom of that bowl-shaped error surface by taking small steps downhill; least squares provides the specific shape that makes this process reliable and fast in many cases. Second, **polynomial regression** extends this idea to fit curves instead of straight lines, using the same principle of minimizing squared errors but allowing for more complex shapes.


## The mathematics

Least squares regression is a method for drawing a straight line through a set of data points so that the line fits the data as closely as possible. It does not require calculus or advanced algebra. It only requires addition, multiplication, and solving simple equations.

**The Goal**
You have a list of pairs of numbers. Each pair represents one observation. For example, if you are predicting house prices, each pair is (size_of_house, price). Let us call the size x and the price y.

You want to find a straight line described by the equation:
y = m * x + b

Here:
m is the slope of the line (how much price changes for each unit of size).
b is the intercept (the predicted price when size is zero).

**The Error**
For any single point (x_i, y_i), the line gives a prediction. We call this prediction ŷ_i (read 'y-hat sub i').
y_hat_i = m * x_i + b

The error for this point is the difference between the actual price and the predicted price:
error_i = y_i - y_hat_i

If we just added up the errors, positive errors (overestimates) would cancel out negative errors (underestimates). To avoid this, we square each error. Squaring makes every number positive and penalizes large mistakes more than small ones.
squared_error_i = (y_i - y_hat_i)^2

**The Cost Function**
The total cost of the line is the sum of the squared errors for all n points in your dataset:
S(m, b) = (y_1 - m*x_1 - b)^2 + (y_2 - m*x_2 - b)^2 + ... + (y_n - m*x_n - b)^2

We want to choose m and b so that S(m, b) is as small as possible.

**Finding the Best Line Without Calculus**
You do not need derivatives to find the best line. You can use a system of two simple linear equations called the 'normal equations'. These equations are derived from the geometry of minimizing the sum of squares, but you can just use them directly.

First, calculate these five sums from your data:
1. n: the number of points.
2. Sum_x: the sum of all x values (x_1 + x_2 + ... + x_n).
3. Sum_y: the sum of all y values (y_1 + y_2 + ... + y_n).
4. Sum_xy: the sum of the products of each pair (x_1*y_1 + x_2*y_2 + ... + x_n*y_n).
5. Sum_x_squared: the sum of the squares of all x values (x_1^2 + x_2^2 + ... + x_n^2).

Then, calculate m and b using these formulas:
m = (n * Sum_xy - Sum_x * Sum_y) / (n * Sum_x_squared - (Sum_x)^2)
b = (Sum_y - m * Sum_x) / n

Note: The denominator for m is (n * Sum_x_squared - (Sum_x)^2). If this value is zero, all your x values are the same, and you cannot draw a unique line.

**Why This Works**
The formula for m balances two things: how much x and y vary together (covariance) and how much x varies on its own (variance). The formula for b ensures the line passes through the center of the data (the average x, average y point).


## A worked example

We have three data points: (1, 2), (2, 3), (3, 5).

Step 1: List the values.
x_1 = 1, y_1 = 2
x_2 = 2, y_2 = 3
x_3 = 3, y_3 = 5
Number of points n = 3.

Step 2: Calculate the five required sums.
Sum_x = 1 + 2 + 3 = 6
Sum_y = 2 + 3 + 5 = 10
Sum_xy = (1*2) + (2*3) + (3*5) = 2 + 6 + 15 = 23
Sum_x_squared = (1^2) + (2^2) + (3^2) = 1 + 4 + 9 = 14
(Sum_x)^2 = 6^2 = 36

Step 3: Calculate the slope m.
Numerator for m = n * Sum_xy - Sum_x * Sum_y
= 3 * 23 - 6 * 10
= 69 - 60
= 9

Denominator for m = n * Sum_x_squared - (Sum_x)^2
= 3 * 14 - 36
= 42 - 36
= 6

m = Numerator / Denominator = 9 / 6 = 1.5

Step 4: Calculate the intercept b.
b = (Sum_y - m * Sum_x) / n
= (10 - 1.5 * 6) / 3
= (10 - 9) / 3
= 1 / 3
b ≈ 0.333

Step 5: Write the final line.
y = 1.5x + 0.333

Step 6: Verify the fit by calculating the squared errors for each point.
Point 1 (1, 2):
Prediction = 1.5(1) + 0.333 = 1.833
Error = 2 - 1.833 = 0.167
Squared Error = 0.167^2 ≈ 0.028

Point 2 (2, 3):
Prediction = 1.5(2) + 0.333 = 3.333
Error = 3 - 3.333 = -0.333
Squared Error = (-0.333)^2 ≈ 0.111

Point 3 (3, 5):
Prediction = 1.5(3) + 0.333 = 4.833
Error = 5 - 4.833 = 0.167
Squared Error = 0.167^2 ≈ 0.028

Total Sum of Squared Errors = 0.028 + 0.111 + 0.028 = 0.167

Result: The least squares line is y = 1.5x + 0.333, with a total sum of squared errors of approximately 0.167.


## From scratch, in Python

```python
# No numpy, just plain Python lists and loops.
# Data points: x and y values
x_vals = [1, 2, 3]
y_vals = [2, 3, 5]

# Number of points
n = len(x_vals)

# Compute sums needed for the formula
sum_x = 0
sum_y = 0
sum_xy = 0
sum_x2 = 0

for i in range(n):
    x = x_vals[i]
    y = y_vals[i]
    sum_x += x
    sum_y += y
    sum_xy += x * y
    sum_x2 += x * x

# Calculate the slope m and intercept b using the normal equations
m = (n * sum_xy - sum_x * sum_y) / (n * sum_x2 - sum_x * sum_x)
b = (sum_y - m * sum_x) / n

print("Slope m =", m)
print("Intercept b =", b)

# Optional: compute the total squared error to confirm
sse = 0
for i in range(n):
    y_pred = m * x_vals[i] + b
    error = y_vals[i] - y_pred
    sse += error * error
print("Sum of squared errors =", sse)
```


## The same thing, with a framework

In NumPy, you can do the same with `np.polyfit(x, y, 1)`, which gives slope and intercept directly. Underneath, it solves the same normal equations. For example:

```python
import numpy as np
x = np.array([1, 2, 3])
y = np.array([2, 3, 5])
m, b = np.polyfit(x, y, 1)
print(m, b)
```


## What the code does

We start with two lists: `x_vals` and `y_vals` that hold our data points. A list in Python is an ordered collection that can be changed. We use the `range(n)` function to loop over indices from 0 to n-1. Inside the loop, we pick out each x and y, and accumulate (add up) the sums we need: sum of x, sum of y, sum of x times y, and sum of x squared. After the loop, we plug those sums into the formulas for m and b (which we derived from setting the derivatives to zero). Finally, we test the line by computing each prediction, the difference to the actual y, squaring it, and adding up — that's the sum of squared errors. The output will show approximately m=1.667 and b=0.


## Common misconceptions

**Least squares is the same as 'fitting a line through all points'** — The line must go through every data point.

With more than two points and noise, no line goes through all exactly; least squares finds the closest compromise. In fact: Least squares finds the line that minimizes total squared error, not one that necessarily touches any point.

**Squaring means we always prefer small errors, but it's arbitrary** — We could just as easily use absolute errors — why square at all?

Squaring gives a unique solution that is mathematically convenient (differentiable) and corresponds to Gaussian noise, which is why it's standard. In fact: Squaring is a choice with good properties: it's smooth, gives a closed-form solution, and matches maximum likelihood under normal errors.

**Least squares is just for lines** — It only works for fitting a straight line.

It extends to polynomial and multi-variable regression by adding more parameters. In fact: Any model that is linear in its parameters can be fit with least squares, including curves.

**The intercept is optional and can be ignored** — If I just fit slope and don't care about intercept, I can set it to zero.

Without an intercept, the line is forced through the origin, which usually makes the fit worse. In fact: Always include the intercept unless you have a strong reason to force the line through zero.


## In short

Least squares regression finds the line (or curve) that minimizes the sum of squared vertical differences from the data. This is a core idea in optimization and appears everywhere in machine learning, from linear regression to neural network loss functions. It has a closed-form solution (direct formula) and a geometric interpretation as projecting data onto a line. Its weakness is sensitivity to outliers, which is why we sometimes use robust alternatives like the Huber loss.


## Where this is used

Least squares is everywhere in machine learning and AI, but often under a different name: 'squared error loss'. When you train a neural network to predict house prices, the loss function is often the mean squared error (average of squared residuals). When you fine-tune a large language model on a regression task (like estimating a difficulty score), the same thing happens. On small devices, tiny models that predict sensor readings use it too.

In retrieval-augmented generation (RAG), the retrieval step often uses embeddings — a list of numbers for each document. When we need to find a similar document, we compute the cosine similarity (which is related to least squares) to rank them.

In agentic systems that plan and call tools, a model might predict how many steps a task will take, or how much a tool call will cost — again, a regression with squared error loss.

More broadly, a developer will use `sklearn.linear_model.LinearRegression` which under the hood uses least squares (often with a solver like SVD). But the plain-vanilla code is what we wrote above.

When done badly — for example, using least squares without checking for outliers, or with features on vastly different scales — the model can be pulled toward bad points. This is why we sometimes use regularized versions (Ridge, Lasso) that add a penalty to keep the coefficients small.


---

Read this on the web: https://basavapurushottam.com/learn-ai/least-squares-regression
