# Linear Regression

Fitting a linear relationship between predictors and a continuous outcome, the foundational model of applied statistics.

**Rests on:** Descriptive Statistics, Least Squares Regression, Normal Equations

**Leads to:** Logistic Regression, Market Risk, Perceptron and Decision Boundaries, Polynomial Regression, Ridge and Lasso Regression, Stock Price Prediction, Support Vector Machines, Two-Class and Multiclass Classifiers

*Machine Learning · Machine Learning Foundations*


## Why learn this

You use linear regression when you need to predict a number based on other numbers, such as estimating next month’s sales from advertising spend or guessing a house’s price from its square footage. It is the simplest way to see how changes in your inputs lead to changes in your output. For example, if you have data showing that each additional bedroom adds roughly $20,000 to a home’s value, linear regression gives you that clear, direct relationship. Mastering it allows you to interpret exactly which factors matter and by how much, rather than treating the prediction as a black box.


## The intuition

Imagine you want to predict the price of a house. You look at three numbers for each past sale: the size in square feet, the number of bedrooms, and the age of the home. Linear regression asks: 'What is the steady, straight-line pattern that connects these inputs to the final price?'

For just one input, like size, you draw a straight line through the dots on a graph where the x-axis is size and the y-axis is price. This line might show that for every extra 100 square feet, the price goes up by $15,000. 

When you have multiple inputs (size, bedrooms, age), the 'line' becomes a flat sheet or surface stretching through the data in many dimensions. We call this shape a hyperplane. It is simply a flat surface that exists even when you can’t easily visualize more than three dimensions. The goal is to find that specific flat surface that comes as close as possible to all the past house prices.

How do we define 'as close as possible'? We look at the vertical distance between each actual sold price and where our flat surface predicted it would be. This difference is the error. To avoid positive errors canceling out negative ones, and to punish large mistakes more severely, we square each of these distances (multiply the error by itself). Then, we add up all those squared errors for every house in your data. The 'best' hyperplane is the one that makes this total sum of squared errors as small as possible.


## An analogy

Think of a waiter balancing a long tray with multiple plates on it. Each plate represents an input factor: one plate holds the weight of 'size', another holds 'bedrooms', and another holds 'age'. The waiter’s arms act as the pivot points, adjusting how much each plate contributes to the final tilt of the tray. The total tilt of the tray corresponds to the predicted price. 

To train the waiter, you give them hundreds of examples of actual houses and their real prices. They adjust the position of the plates (the coefficients) until, across all those examples, the tray’s tilt matches the actual weight of the house as closely as possible. 

The analogy holds because both involve balancing contributions to reach a total. It breaks down only in that a waiter physically feels gravity and balance, whereas linear regression mathematically calculates positions to minimize distance, allowing for negative contributions (like older houses lowering the price) which are harder to visualize on a physical tray.


## Definition

Linear regression is a method of finding the flat surface that best fits historical data by adjusting its angles so that the total squared difference between its predictions and the actual values is as small as possible.


## How it connects

This concept serves as the foundational grammar for many other prediction models. Its neighboring topics include Logistic Regression, which uses the same flat-surface logic but squashes the result to predict a yes-or-no outcome instead of a number, and Polynomial Regression, which allows the surface to curve rather than stay flat to capture more complex relationships.


## The mathematics

This concept rests on two branches of mathematics: arithmetic — which you already know — and a piece of calculus. The core operation is multiplication and addition: to produce a prediction, multiply each input by its own coefficient, add all those products, then add an intercept. That is the entire model. The mathematics of how to choose the best coefficients is where it goes slightly beyond arithmetic. Finding the best line is a problem of minimisation: you have a quantity (the sum of squared errors) that changes as you change the slope and intercept. You want the pair of numbers that makes it as small as possible. This is the branch of mathematics called calculus — specifically, finding the minimum of a function by setting its derivative to zero. You have never met a derivative, so here it is in words: the derivative of a function at a point tells you the steepness of the graph at that point. If the graph is sloping down, you can move right to decrease; if up, move left. The minimum is exactly where the slope is flat — zero. For the sum of squared errors, there is a single flat point, and it can be found directly by solving a small set of equations. That is what your code will do under the hood. You will not need to calculate a derivative by hand this time, but you will see its shadow: the path from the formula for the error to the formula for the best coefficients passes through that exact idea — finding where the slope is zero.

For a single input x and continuous output y, the model is:

ŷ = w·x + b

where w is the slope, b is the intercept, and ŷ (read 'y-hat') is the predicted value.

The error for one point is (y − ŷ). The sum of squared errors (SSE) is:

SSE = Σᵢ (yᵢ − ŷᵢ)²

The best w and b are the ones that minimise SSE. There is a direct formula (using derivatives), but you will approximate it with iterative code.


## A worked example

```
Three past house sales (area in square feet, price in ₹ lakh):
Sale A: 800 sq ft, ₹40 lakh
Sale B: 1000 sq ft, ₹50 lakh
Sale C: 1200 sq ft, ₹62 lakh
We want the best straight line: price = (slope × area) + intercept.

Step 1: Guess: slope = 0.05, intercept = 0. Then predicted prices: 40, 50, 60.   # For each sale, multiply area by 0.05 and add 0. For 800: 0.05×800 = 40; for 1000: 50; for 1200: 60.
Step 2: Errors: Sale A 40−40 = 0; Sale B 50−50 = 0; Sale C 62−60 = 2.   # Compare predicted to actual.
Step 3: Squared errors: 0², 0², 2² = 4. Sum = 4.   # Square each error and add.
Step 4: Try slope = 0.06, intercept = 0. Predictions: 48, 60, 72.   # A different guess — you try many to find the minimum.
Step 5: Errors: 40−48 = −8; 50−60 = −10; 62−72 = −10.   # Negative here means over-prediction.
Step 6: Squared errors: 64 + 100 + 100 = 264. Worse than 4.   # So slope 0.05 is better than 0.06 for this intercept.

Result: The true best line, found by the calculus method, is: price = 0.055 × area − 4.0. Check: for 800 → 40; for 1000 → 51; for 1200 → 62. The sum of squared errors is 2, smaller than both guesses. The process of trying values and comparing sums of squared errors is the essence of gradient descent, which you will use later.
```


## From scratch, in Python

```python
# Linear regression from scratch, no libraries.

# The data: areas (sq ft) and prices (lakh ₹).
areas = [800, 1000, 1200]
prices = [40, 50, 62]

# A function to compute predictions given slope and intercept.
def predict(area, slope, intercept):
    return slope * area + intercept

# A function to compute the sum of squared errors.
def sum_squared_errors(slope, intercept):
    total = 0
    for i in range(len(areas)):
        y_hat = predict(areas[i], slope, intercept)
        error = prices[i] - y_hat
        total = total + error * error
    return total

# Try a few slopes with intercept 0.
intercept = 0.0
best_slope = 0
best_sse = None

for slope in [0.04, 0.05, 0.055, 0.06, 0.07]:
    sse = sum_squared_errors(slope, intercept)
    print("slope", slope, "SSE", sse)
    if best_sse is None or sse < best_sse:
        best_sse = sse
        best_slope = slope

print("Best slope seen:", best_slope, "with SSE", best_sse)

# Simple gradient descent: move the slope downhill.
# We estimate the derivative numerically.
slope = 0.0
intercept = 0.0
learning_rate = 0.000001

for step in range(1000):
    h = 0.001
    sse_current = sum_squared_errors(slope, intercept)
    sse_slightly_higher = sum_squared_errors(slope + h, intercept)
    derivative = (sse_slightly_higher - sse_current) / h
    slope = slope - learning_rate * derivative

print("After 1000 steps, slope =", slope)
```


## The same thing, with a framework

```python
import numpy as np

# Data as arrays.
X = np.array([800, 1000, 1200])
y = np.array([40, 50, 62])

# Fit a line: this is the closed-form solution (least squares).
# It computes the slope and intercept exactly, using linear algebra.
slope, intercept = np.polyfit(X, y, 1)
print("Slope:", slope, "Intercept:", intercept)

# Or using the formula: slope = covariance(X, y) / variance(X)
slope_formula = np.cov(X, y, ddof=0)[0, 1] / np.var(X)
intercept_formula = np.mean(y) - slope_formula * np.mean(X)
print("Slope via formula:", slope_formula, "Intercept:", intercept_formula)
```


## What the code does

First, the lists. `areas = [800, 1000, 1200]` is Python's way of storing an ordered collection of numbers — a list keeps its order, and you can change it later. The first area corresponds to the first price, which is why lists are useful here. `len(areas)` is 3, so `range(len(areas))` gives the numbers 0, 1, 2. The `for i in range(...)` loop runs the block below it once for each of those numbers, with `i` taking each value in turn. Inside, `areas[i]` picks out the element at position i — counting starts at zero, so `areas[0]` is 800, `areas[1]` is 1000, `areas[2]` is 1200. The function `def predict(area, slope, intercept):` defines a reusable block. It takes three inputs and returns one output: `slope * area + intercept`. That is exactly the equation ŷ = w·x + b — the model. The `sum_squared_errors` function loops over all data, computes the prediction, subtracts from the actual price to get the error, squares it, and adds to a running total. That is the SSE. A key idea: `best_sse = None` — a variable that holds 'no value yet'. The `if best_sse is None or sse < best_sse:` tests first whether we have no best yet, and if so takes the first value; otherwise it compares. This is the pattern for finding a minimum by brute force. The final loop is gradient descent: it estimates the derivative of SSE with respect to slope using a small step `h`, then moves the slope in the opposite direction (downhill) by a small amount `learning_rate`. Repeating many times walks toward the minimum. The framework version does the same but with a famous closed-form formula — no iteration needed — using NumPy's built-in functions, which under the hood are doing exactly the multiplication and addition you did by hand.


## Common misconceptions

**Correlation implies causation** — If linear regression finds a strong relationship and a positive coefficient, then changing the input will change the output.

The model only measures association. It says nothing about cause. A third hidden variable could drive both, or the direction could even be reversed. In fact: Regression finds relationships in data; causality requires a controlled experiment or a careful causal model. A coefficient being non-zero does not mean you can act on it.

**The line must pass through the points** — A good regression line should go through every data point. If it does not, the model is wrong.

Real data has noise. Forcing the line through all points overfits — it captures the noise, not the pattern. A line that is close on average is better than one that hits each point exactly. In fact: The goal is to minimise the sum of squared errors across all points, not to equal them individually. The line is a summary, not a perfect reproduction.

**Bias is unfairness** — The intercept term 'bias' in the model is statistical bias — it makes the model unfair or prejudiced.

In this context, 'bias' is a technical term for the intercept — the prediction when all inputs are zero. It has nothing to do with social bias. In fact: Here, bias is just the constant term b in ŷ = w·x + b. It shifts the line up or down. Societal bias in AI is a separate, important topic.

**Linear regression is only for straight lines** — If my data curves, linear regression is useless; I need a neural network.

You can add polynomial terms (x², x³) to the inputs, and the model is still 'linear in the parameters' — meaning the mathematics does not change. More generally, a linear model is a good baseline even for curved data if you transform the inputs first. In fact: Linear regression can fit curves by including powers of predictors. It remains the first model to try; a complex model is only justified if the linear baseline fails validation.

**A large R² means the model is good** — If the R-squared (the proportion of variance explained) is high, the model must be excellent.

R² can be inflated by overfitting, by adding many irrelevant predictors, by extrapolating beyond the data range, or by using too few points. It says nothing about whether predictions are accurate on new data. In fact: R² describes fit to the training data. The real test is performance on hold-out data. Always validate on unseen data.

**The error is a mistake I made** — If the model's prediction is not exactly the actual price, I must have coded it wrongly.

No — residual error is a fundamental part of any statistical model. The world is noisy; even a perfect relationship has irreducible error. The model's job is to capture the systematic part. In fact: A model predicts the expected value, not the exact outcome. The residuals are the leftover noise, which you expect to be small but not zero.


## In short

Linear regression finds the straight line (or flat plane in many dimensions) that best fits data, measured by the sum of squared errors. Each input gets a coefficient (its slope) and there is an intercept. The 'fit' is a search for coefficients that minimise that error. It is the first model in your machine learning path: interpretable, fast, and the right starting point for any prediction problem. Remember: correlation is not causation, residuals are normal, and validation on unseen data is the only real test.


## Where this is used

Linear regression appears everywhere in AI, though often wearing different clothes.

In large language models: The transformer architecture that powers GPT-class models uses linear operations at every step. Every 'linear layer' or 'dense layer' that sits between attention heads, every projection of a token embedding, every readout that maps hidden states to a vocabulary — all are exactly the operation you learned today: input numbers multiplied by weights, summed, plus a bias. Before any of that complexity, an engineer might also use linear regression to forecast compute requirements (next month's GPU hours vs. model size), to predict latency from sequence length, or to calibrate output probabilities.

In small language models running on devices: When a model must fit in a few megabytes, engineers often use knowledge distillation — training a small student model on a large teacher's outputs. A simple calibration step fits a linear mapping between the teacher's logits and the student's. Also, quantization — reducing the precision of weights — often starts by fitting a linear regression that maps float weights to their 8-bit versions with minimal loss.

In retrieval-augmented generation (RAG): RAG retrieves documents relevant to a query, and relevance is typically scored. A common method is to fit a linear regression — or a simple ranking model — that combines features like cosine similarity, term frequency, and document length into a single relevance score. The 'embedding and similarity machinery' you noted in your library uses dot products for similarity; a linear regression head on top of those embeddings is how many systems decide what to fetch. The coefficients tell you which similarity measures actually matter.

In agentic systems that plan and call tools: An agent that decides which tool to call often scores each candidate action with a learned function. A logistic regression (a linear combination squashed through a sigmoid) is the simplest version; linear regression scores continuous outcomes — how much time will this action save, how likely is this tool to succeed. In planning, uncertainty estimates from regression models help decide when to stop and ask for help.

In everyday applied machine learning: This is where linear regression shines. Churn prediction: a telecom may predict the expected days until a customer leaves, with inputs of usage and complaints. Crop disease detection: a regression can predict disease severity from leaf colour values before a classifier decides. Forecasting: predicting next month's demand from season and past sales. Fraud detection: scoring a transaction's probability of being fraudulent (though logistic regression is used, it is built on the same foundation). Any spreadsheet analyst fitting a trendline is doing linear regression.

What a developer actually writes: In Python, they call `from sklearn.linear_model import LinearRegression`, then `model.fit(X, y)` where X is a table of inputs (a 2D array) and y is a list of outputs. That one call does the same minimisation you did by hand. What breaks when done badly: not checking the assumptions — if the true relationship is curved, or the errors grow with the size of the prediction, the line is misleading. The coefficients then mislead business decisions. Another common failure: extrapolating — predicting for inputs far outside the range of the training data; the line keeps going straight where reality bends. Developers must validate on data never seen, and plot the residuals (errors) against predictions to spot problems.


---

Read this on the web: https://basavapurushottam.com/learn-ai/linear-regression
