# Polynomial Regression

Extending linear models with powers of the predictors to fit curvature while remaining linear in parameters.

**Rests on:** Least Squares Regression, Linear Regression

*Machine Learning · Machine Learning Foundations*


## Why learn this

Polynomial regression is the first step beyond straight lines. Once you master it, you will understand why machine learning models can bend and curve to fit data—and why too much bending is a trap. This concept unlocks the study of overfitting, model evaluation, and the bias-variance tradeoff, which are the gates to every serious modelling technique that follows: splines, regularisation (ridge and lasso), and even the multi-layer neural networks behind modern AI.


## The intuition

So far, you have probably fit straight lines to data: something like 'for every extra year of experience, salary rises by ₹50,000.' But the world is not always straight. Prices rise, then fall. Growth accelerates, then slows. A straight line cannot capture a U-shape or an S-curve. Polynomial regression lets the line bend by adding powers of the input: the input squared, cubed, and so on. Instead of drawing just y = a + b*x, we draw y = a + b*x + c*x² + d*x³. Each power adds one curve. The first power is the straight trend, the second power adds a single bend (like a valley or a hill), the third adds an S-curve, and higher powers add more wiggles. With enough powers, you can draw nearly any shape—but there is a price: a curve that wiggles too much will fit the training data perfectly and fail on new data. That is the central tension of this concept: flexibility versus reliability.


## An analogy

Imagine you are tracing the outline of a leaf with a pencil. A straight line from the base to the tip would capture the general direction but none of the leaf's curves. Your first attempt uses one straight line: it is simple, but it misses most of the shape. So you try a curved stroke—the leaf's outline bends gently, and you get closer. But the leaf has wiggles, so you keep adding gentle curves to your stroke: a little bend here, a small wave there. Now the outline follows the leaf almost perfectly. However, if the leaf has a dirt speck or a torn edge, your pencil will faithfully trace that tiny imperfection too. If you then try to draw a different leaf from the same species, your overly detailed outline will fail because it has copied the speck of the first leaf, not the general leaf shape. Polynomial regression works exactly this way: adding powers is like adding curves to your pencil stroke. Low degree: straight line, too simple. Medium degree: captures the main bends, works well on new leaves. Very high degree: traces every speck, fails on anything new.


## Definition

Polynomial regression is an extension of linear regression that predicts an outcome using powers of the input (like x², x³) as extra predictors, so the fitted curve can bend, while still being a linear model in the parameters (the coefficients being learned).


## How it connects

This concept builds directly on Linear Regression, which you have already studied. In linear regression, you fitted a straight line: y = a + b*x. Here, you will do the same arithmetic but with extra columns for x², x³, etc. The parameters (a, b, c, ...) are still found by the same least-squares method. The change is purely in the input features, not in the learning algorithm. This also connects to your notes on Ridge and Lasso Regression, which are the standard ways to control the wild bends that high-degree polynomials introduce. If you go on to study Splines (mentioned in your notes), you will see that they achieve the same flexibility as high-degree polynomials but with much better stability.


## The mathematics

Polynomial regression is a way of drawing a curved line through data points. It works by turning a curve-fitting problem into a straight-line fitting problem. This might sound contradictory, but it relies on two simple ideas: treating powers of x as separate variables, and using the same arithmetic for finding coefficients that we use for linear regression.

**1. The Model: Turning Curves into Sums**

In ordinary linear regression, we assume the relationship between an input x and an output y is a straight line:
y = a + b*x

Here, 'a' is the starting height (intercept) and 'b' is the slope. The word "linear" means that the coefficients a and b appear only as single numbers multiplied by x or 1. They are not squared, divided, or inside other functions.

Polynomial regression keeps this structure but adds more terms. Instead of just x, we also use x squared (x²), x cubed (x³), and so on. A polynomial of degree n looks like this:
y = a₀ + a₁*x + a₂*x² + ... + aₙ*xⁿ

Let us break down what each part means:
- y is the value we are trying to predict.
- x is the input number we start with.
- a₀, a₁, ..., aₙ are the coefficients. These are the numbers we need to find. They determine how much weight each term (1, x, x², etc.) contributes to the final answer.
- The powers of x (x, x², x³) are called features or basis functions. Even though x² is a curved shape, in the equation above, it acts just like another variable. If we call x¹ = v₁ and x² = v₂, the equation becomes y = a₀ + a₁*v₁ + a₂*v₂. This is still linear in the coefficients a₀, a₁, a₂.

**2. Why "Linear"?**

The term "linear polynomial regression" can be confusing because the resulting graph is curved. The linearity refers to the coefficients, not the input x. Because each coefficient (a₀, a₁, etc.) is multiplied by a known number (1, x, x²) and added, the math for finding these coefficients remains linear algebra. We do not need to solve complex nonlinear equations; we only need to solve a system of linear equations.

**3. Finding the Coefficients: Least Squares**

To find the best values for a₀, a₁, ..., aₙ, we use a method called Ordinary Least Squares. The goal is to make the predicted y values as close as possible to the actual observed y values in our data.

Here is the step-by-step arithmetic process:
1. For each data point i (where i goes from 1 to N, the total number of points), calculate the prediction using the current coefficients:
   Prediction_i = a₀ + a₁*x_i + a₂*(x_i)² + ... + aₙ*(x_i)ⁿ
2. Calculate the error for that point:
   Error_i = Actual_y_i - Prediction_i
3. Square the error to make it positive and penalize large errors more heavily:
   Squared_Error_i = (Error_i)²
4. Add up all the squared errors across all data points. This total is called the Sum of Squared Errors (SSE):
   SSE = Σ (from i=1 to N) [Actual_y_i - Prediction_i]²
5. The "best" coefficients are the specific set of numbers a₀, a₁, ..., aₙ that make this SSE as small as possible.

In practice, we do not guess these numbers. We use linear algebra (specifically, solving the normal equations or using matrix operations) to find the exact minimum. The key takeaway is that the complexity of the curve comes from the powers of x, but the method for finding the weights remains the standard linear regression algorithm.

**4. Overfitting and Model Choice**

As we increase the degree n (adding more terms like x⁴, x⁵), the polynomial becomes more flexible. It can wiggle more to pass through every data point exactly. However, this flexibility has a cost. If the degree is too high relative to the amount of data, the model may fit the noise in the training data rather than the underlying pattern. This is called overfitting. The reference passages note that overfitting occurs when the model class is "too rich" for the dataset. A lower-degree polynomial might generalize better to new, unseen data, even if it has a slightly higher error on the training data.


## A worked example

Let us walk through the arithmetic of making predictions with a specific polynomial, using the same data points as before: (1, 3), (2, 6), and (3, 11).

We will fit a quadratic polynomial (degree 2):
y = a + b*x + c*x²

**Step 1: Choose trial coefficients**

To see how the prediction works, let us pick arbitrary numbers for the coefficients:
a = 1
b = 2
c = 1

So our prediction formula is:
y_predicted = 1 + 2*x + 1*x²

**Step 2: Calculate predictions for each data point**

*Point 1: x = 1, Actual y = 3*
- Calculate the terms:
  - Constant term: a = 1
  - Linear term: b*x = 2 * 1 = 2
  - Quadratic term: c*x² = 1 * (1)² = 1 * 1 = 1
- Sum them up:
  y_predicted = 1 + 2 + 1 = 4
- Calculate the error:
  Error = Actual y - Predicted y = 3 - 4 = -1
- Square the error:
  Squared Error = (-1)² = 1

*Point 2: x = 2, Actual y = 6*
- Calculate the terms:
  - Constant term: a = 1
  - Linear term: b*x = 2 * 2 = 4
  - Quadratic term: c*x² = 1 * (2)² = 1 * 4 = 4
- Sum them up:
  y_predicted = 1 + 4 + 4 = 9
- Calculate the error:
  Error = Actual y - Predicted y = 6 - 9 = -3
- Square the error:
  Squared Error = (-3)² = 9

*Point 3: x = 3, Actual y = 11*
- Calculate the terms:
  - Constant term: a = 1
  - Linear term: b*x = 2 * 3 = 6
  - Quadratic term: c*x² = 1 * (3)² = 1 * 9 = 9
- Sum them up:
  y_predicted = 1 + 6 + 9 = 16
- Calculate the error:
  Error = Actual y - Predicted y = 11 - 16 = -5
- Square the error:
  Squared Error = (-5)² = 25

**Step 3: Sum the squared errors**

Total SSE = 1 + 9 + 25 = 35

This total of 35 tells us how far off our trial coefficients (a=1, b=2, c=1) are from the data. A lower number would be better.

**Step 4: Finding the true coefficients**

The text previously mentioned that the "real" coefficients for these three points might be a=1, b=-1, c=2. Let us verify this by plugging them into our formula:
y = 1 - 1*x + 2*x²

*Point 1: x = 1*
- y_predicted = 1 - 1(1) + 2(1)² = 1 - 1 + 2 = 2
- Actual y = 3
- Error = 3 - 2 = 1
- Squared Error = 1

*Point 2: x = 2*
- y_predicted = 1 - 1(2) + 2(2)² = 1 - 2 + 8 = 7
- Actual y = 6
- Error = 6 - 7 = -1
- Squared Error = 1

*Point 3: x = 3*
- y_predicted = 1 - 1(3) + 2(3)² = 1 - 3 + 18 = 16
- Actual y = 11
- Error = 11 - 16 = -5
- Squared Error = 25

Total SSE = 1 + 1 + 25 = 27.

Wait, the previous text claimed a perfect fit (error 0) with different coefficients. Let us solve for the exact least squares solution for these three points to ensure accuracy. With 3 points and 3 coefficients (a, b, c), a degree 2 polynomial can pass through all 3 points exactly if they are not collinear in a way that prevents it (which they don't here). 

Let us find the exact coefficients that make SSE = 0.
We need:
1) a + b(1) + c(1)² = 3  => a + b + c = 3
2) a + b(2) + c(4) = 6   => a + 2b + 4c = 6
3) a + b(3) + c(9) = 11  => a + 3b + 9c = 11

Subtract equation (1) from (2):
b + 3c = 3  => b = 3 - 3c
Subtract equation (2) from (3):
b + 5c = 5

Substitute b into the second result:
(3 - 3c) + 5c = 5
3 + 2c = 5
2c = 2
c = 1

Then b = 3 - 3(1) = 0.
Then a = 3 - b - c = 3 - 0 - 1 = 2.

So the exact coefficients are a=2, b=0, c=1.
The polynomial is y = 2 + x².

Let us check this perfect fit:
- x=1: y = 2 + 1 = 3. (Matches actual 3)
- x=2: y = 2 + 4 = 6. (Matches actual 6)
- x=3: y = 2 + 9 = 11. (Matches actual 11)

Total SSE = 0.

This example demonstrates that with enough coefficients (degree n >= number of points - 1), we can fit the training data perfectly. However, as noted in the theory section, this perfect fit may not generalize well to new data if the underlying relationship is simpler.


## From scratch, in Python

```python
# We'll implement polynomial regression from scratch in plain Python.
# First, let's teach ourselves the required Python building blocks.

# A list is an ordered collection, like a row of data. We can have numbers in it.
x_values = [1, 2, 3]  # three data points for x

y_values = [3, 6, 11]  # matching y values

# A for loop lets us go through each element of a list, one by one.
# Here we print each x value and its square.
for x in x_values:
    print("x:", x, "x squared:", x**2)

# We will build a feature matrix: each row is one data point, and the columns are [1, x, x^2, ...].
# degree is 2, so we have columns for x^0 (=1), x^1, x^2.
degree = 2

# Start with an empty list to hold all rows.
X = []

for x in x_values:
    row = []
    for power in range(degree + 1):  # powers 0, 1, 2
        row.append(x**power)
    X.append(row)

print("Feature matrix X:")
for row in X:
    print(row)

# Now, we need to find the coefficients a0, a1, a2 that make predictions close to y.
# We'll use the method of normal equations: a = (X^T X)^-1 X^T y
# We need matrix operations, so we'll implement transpose and multiplication by hand.

def transpose(matrix):
    # If matrix has rows and columns, new matrix has those swapped.
    rows = len(matrix)
    cols = len(matrix[0]) if rows > 0 else 0
    # Initialize empty matrix with cols rows and rows columns.
    result = [[0 for _ in range(rows)] for _ in range(cols)]
    for i in range(rows):
        for j in range(cols):
            result[j][i] = matrix[i][j]
    return result

def matmul(A, B):
    # A is m x p, B is p x n, result is m x n.
    m = len(A)
    p = len(A[0]) if m > 0 else 0
    n = len(B[0]) if len(B) > 0 else 0
    result = [[0 for _ in range(n)] for _ in range(m)]
    for i in range(m):
        for j in range(n):
            total = 0
            for k in range(p):
                total += A[i][k] * B[k][j]
            result[i][j] = total
    return result

def invert_2x2(matrix):
    # For degree up to 2, we get a 3x3 matrix. Use general formula? That''s complex.
    # For simplicity, we could use numpy, but we are writing from scratch.
    # Let's implement a generic matrix inversion using Gauss-Jordan.
    n = len(matrix)
    # Augment with identity matrix.
    augmented = [row[:] + [1 if i == j else 0 for j in range(n)] for i, row in enumerate(matrix)]
    for i in range(n):
        # Make sure pivot is not zero.
        if augmented[i][i] == 0:
            # Swap with a row below that has non-zero pivot.
            for k in range(i+1, n):
                if augmented[k][i] != 0:
                    augmented[i], augmented[k] = augmented[k], augmented[i]
                    break
        pivot = augmented[i][i]
        if pivot == 0:
            raise ValueError("Singular matrix")
        # Scale row to make pivot 1.
        for j in range(2*n):
            augmented[i][j] /= pivot
        # Eliminate other rows.
        for k in range(n):
            if k != i:
                factor = augmented[k][i]
                for j in range(2*n):
                    augmented[k][j] -= factor * augmented[i][j]
    return [row[n:] for row in augmented]

# Compute X^T X and X^T y.
XT = transpose(X)
XTX = matmul(XT, X)
XTy = matmul(XT, [[y] for y in y_values])  # each y as a single-element list

# Invert XTX
XTX_inv = invert_2x2(XTX)

# Multiply inverse by XTy to get coefficients.
coefficients = matmul(XTX_inv, XTy)

# Extract as a flat list.
coeff = [row[0] for row in coefficients]
print("Coefficients (a0, a1, a2):", coeff)

# Now predict for a new x value, say x=2.5.
new_x = 2.5
prediction = sum(coeff[p] * (new_x ** p) for p in range(len(coeff)))
print("Prediction for x = 2.5:", prediction)
```


## The same thing, with a framework

```python
import numpy as np

# The same polynomial regression using NumPy's polyfit, which does the least squares for us.
# Underneath, it builds the feature matrix X with powers and solves the normal equations.

x = np.array([1, 2, 3])
y = np.array([3, 6, 11])
degree = 2

# np.polyfit returns coefficients from highest degree down to constant.
coeffs = np.polyfit(x, y, degree)
# To use, we can create a polynomial function:
p = np.poly1d(coeffs)
print("Polynomial coefficients (highest first):", coeffs)
print("Prediction for x=2.5:", p(2.5))

# This is exactly what our from-scratch code did: build X with powers, compute X^T X, invert, etc.
```


## What the code does

The from-scratch code starts with our data: x_values and y_values. Then we create the feature matrix X. Each row corresponds to a data point, and we append the powers of x from 0 up to the degree. We use a for loop and we calculate x**power. For example, for x=2, power 0 gives 1, power 1 gives 2, power 2 gives 4. Then we move on to solve the least squares problem. We use the normal equations method: the coefficients are found by solving (X^T X) * a = X^T y, which is a standard algebraic solution. We had to implement matrix transpose, multiplication, and inversion by hand. The inversion is the most complex part, using Gauss-Jordan elimination. Finally, we get the coefficients and we can make predictions for new inputs. The framework version in NumPy simply calls np.polyfit, which hides all that detail, but the underlying math is exactly the same. np.poly1d lets us evaluate the polynomial.


## Common misconceptions

**Polynomial regression is not a linear model** — Polynomial regression draws curves, so it can't be linear. 'Linear' must mean the equation is a straight line.

In statistics, 'linear' refers to the parameters, not the shape of the curve. The model is linear in the coefficients: y = a₀ + a₁*x + a₂*x². Each coefficient is a constant multiplied by a term. There is no a₁² or a₁*a₂. That is what makes it linear. In fact: Polynomial regression is a linear model because the predicted y is a linear combination of the parameters (a₀, a₁, a₂, ...), even though it is non-linear in the input x.

**Higher degree always fits better** — If degree 2 fits well, degree 3 fits better, and degree 10 is even better. More complexity is always an improvement.

While the sum of squared errors on the training data always decreases (or stays the same) as you add more terms, the model starts to capture noise, not just the pattern. This is called overfitting. The performance on new, unseen data gets worse. In fact: Higher degrees fit the training data better, but they often perform worse on new data. The goal is to minimise error on unseen data, not on the training data you already have.

**Coefficients are interpretable as marginal effects** — In a polynomial model, we can say 'a one-unit increase in x increases y by a₁ units'.

Because x appears in multiple terms (x, x², x³), the effect of a change in x on y depends on the current value of x. The coefficient a₁ is not the total effect; it is only part of it. In fact: The marginal effect of x is the derivative: a₁ + 2*a₂*x + 3*a₃*x² + ... — it changes as x changes. Coefficients in polynomial regression are not individually interpretable in the simple linear way.

**Polynomial regression is outdated** — Nobody uses polynomial regression anymore; they use neural networks or random forests.

Polynomial regression is still widely used in specialised fields like biology and economics, and as a building block in machine learning (e.g., polynomial kernels, feature engineering). It is also a fundamental step for understanding more complex models. In fact: Polynomial regression remains a practical tool for small tabular data, physics-based modelling, and as a starting point for understanding non-linearity and overfitting.

**Polynomial regression can fit any data perfectly** — If I have enough data points, I can always find a polynomial that goes through all of them exactly.

Yes, with a degree equal to (number of points - 1), you can fit exactly, but that is almost never desirable. The polynomial will be very complex and will oscillate wildly between the points, giving terrible predictions for new data. Also, if there is noise, you are fitting the noise, not the signal. In fact: While a polynomial of sufficiently high degree can perfectly interpolate the training data, this generally leads to drastic overfitting and poor generalisation. The goal is a smooth curve that captures the trend, not one that hits every point.


## In short

Polynomial regression extends linear regression by adding powers of the input as features, allowing the curve to bend. It is still linear in the parameters, which means the same least-squares solving methods apply. The key insight is the flexibility–overfitting tradeoff: more power means a better fit to training data but worse predictions on new data. Regularisation and cross-validation are essential tools to control this volatility.


## Where this is used

Polynomial regression might seem like an old-school technique, but its principles are everywhere in modern AI. In large language models (LLMs) like GPT, the 'attention' mechanism uses what is essentially a weighted sum of values, and the transformation layers often include non-linearities that are approximated by polynomial expansions. More directly, in feature engineering for tabular data, practitioners routinely add polynomial features (like x², , x*y) to let linear models capture interactions. For example, in a churn prediction model, the effect of 'usage months' on churn might be non-linear: new users churn a lot, then stay, then churn again after a long time. A polynomial term can capture that U-shape. In small models running on devices, polynomial features are cheap to compute and can replace heavy neural networks for simple tasks like predicting battery drain from usage patterns. In retrieval-augmented generation (RAG), the similarity between embeddings is often a dot product, but some systems use kernel methods that implicitly compute polynomial combinations of the embedding components to capture more complex matches—without explicitly computing the high-dimensional space. Agentic systems that plan and call tools often use polynomial regression as a lightweight way to model rewards or costs over time. For example, an agent deciding when to retry a failed API call might model the success probability as a function of time since last attempt, and a quadratic curve might fit better than a straight line. In everyday applied work, polynomial regression is the go-to when you suspect curvature: forecasting sales with seasonal effects, predicting energy consumption vs. temperature, or modelling the relationship between fertilizer amount and crop yield (which peaks and then declines). Developers typically use scikit-learn's PolynomialFeatures to create the new columns and then LinearRegression to fit, or numpy.polyfit for quick experiments. What breaks when done badly? If you use too high a degree, the model will oscillate wildly at the boundaries, giving absurd predictions for new data. For example, a degree-20 polynomial through ten points might predict a temperature of 10,000°C at x slightly outside the data range. That's why regularisation (ridge or lasso) is important, and why modern practice favours splines. The key takeaway: polynomial regression teaches you the fundamental tradeoff between bias (sticking to a simple shape) and variance (flexibility), which is the core of all model selection.


---

Read this on the web: https://basavapurushottam.com/learn-ai/polynomial-regression
