# Ridge and Lasso Regression

Regularised linear regression: ridge penalises squared coefficients, lasso penalises absolute values.

**Rests on:** Least Squares Regression, Linear Regression

*Machine Learning · Regularization*


## Why learn this

This is the first concept you'll learn that directly controls how well a model performs on new data. Without it, your models will memorize your training data and fail in the real world. Mastering ridge and lasso unlocks the rest of the 'Regularization' topic: dropout, early stopping, data augmentation. From there you'll go to Deep Learning and Model Evaluation. In practice, any serious machine learning you do — from predicting house prices to building a recommendation system — will use these techniques to keep your models honest.


## The intuition

You have a set of data points and you want to draw a line through them that predicts something new. But if you give that line too much freedom — too many knobs to turn — it will twist and bend to touch every single data point, even the noise. That's overfitting: perfect on your training data, useless on anything new. Ridge and Lasso are two ways to pull those knobs back. They add a cost to having large knob settings, so the model learns to keep them small. Think of it as a budget: you have 100 rupees to spend on your knobs. Ridge charges you per rupee you spend, so you buy only the knobs that matter. Lasso charges you per knob you turn on at all, forcing you to switch most of them off entirely. The result is a simpler, more reliable line.


## An analogy

Imagine you're moving to a new apartment and you have to decide which of your 50 boxes to take with you. You have two strategies. Ridge: you can take all 50 boxes, but you pay a storage fee for each kilogram. So you take the heavy ones because they're essential, and you take the light ones but you wish you could leave them behind — you end up with a little bit of everything. Lasso: you have a flat fee per box you take, so you only take the boxes you truly need. The rest stay in storage (their coefficient becomes zero). In machine learning, your 'boxes' are your predictor variables. Ridge shrinks all their influence but keeps them all. Lasso selects a few and discards the rest. But the analogy breaks down: in real moving, you can't take a fraction of a box. In ridge, you actually shrink a coefficient to 0.7 or 0.2 — you keep a little of every variable. And lasso doesn't just pick a box and leave it — it actually forces the coefficient to zero, which is like throwing the box away entirely. So ridge is gentle, lasso is decisive.


## Definition

Ridge and Lasso are regularized linear regression methods that add a penalty to the size of the coefficients: ridge adds the sum of squared coefficients, lasso adds the sum of absolute values, and this penalty forces the coefficients to be smaller (ridge) or zero (lasso).


## How it connects

You haven't learned anything else yet, so this is your gateway. The only math we'll use is arithmetic — adding, subtracting, multiplying, squaring, and taking absolute values. You'll need to understand what a coefficient is (the number that multiplies a variable in a linear equation). If you've ever seen y = 2x + 1, the 2 and 1 are coefficients. We'll extend that idea. Everything later — the full world of machine learning — builds on this foundation.


## The mathematics

We want to draw a line through points on a graph. A line has a formula like y = a*x + b. Here 'a' is the coefficient (how steep the line is) and 'b' is the intercept (where the line crosses the y-axis). Normally, we choose 'a' and 'b' to make the line as close to the points as possible. That is measured by the 'squared error'. For each point, we find the vertical distance from the point to the line. We square that distance so that positive and negative errors do not cancel each other out. Then we add up all those squared distances. This total sum is called the cost. The lower the cost, the better the fit.

If we only minimize this cost, we might get a very wiggly line that fits the training data perfectly but performs poorly on new data. To prevent this, we add a penalty to the cost. This technique is called regularization.

Ridge regression adds a penalty equal to the sum of the squares of all coefficients. It does not penalize the intercept 'b', because shifting the entire line up or down is usually harmless compared to making the slope too steep. The formula for the Ridge cost is:

Cost_ridge = (1/N) * Sum(y_i - y_hat_i)^2 + lambda * Sum(w_j^2)

Lasso regression adds a penalty equal to the sum of the absolute values of all coefficients. The formula for the Lasso cost is:

Cost_lasso = (1/N) * Sum(y_i - y_hat_i)^2 + lambda * Sum(|w_j|)

In these formulas:
- N is the number of data points.
- y_i is the actual output value for the i-th data point.
- y_hat_i is the predicted output value for the i-th data point, calculated using the current coefficients.
- w_j represents each coefficient in the model (excluding the intercept b).
- lambda (lambda) is a positive number that controls the strength of the penalty. It is often written as 'L' or 't' in other texts.

The role of lambda is critical:
- If lambda is zero, the penalty term disappears, and we have plain linear regression (ordinary least squares).
- As lambda increases, the penalty becomes larger. The optimization process must now balance fitting the data well against keeping the coefficients small. This forces the coefficients to shrink toward zero.
- In Lasso, because the penalty uses absolute values, increasing lambda can force some coefficients to become exactly zero. This effectively removes those variables from the model, performing automatic feature selection.

Choosing the right value for lambda is a 'tuning' step. We typically try many values and pick the one that gives the best prediction on unseen data. For now, we assume lambda is fixed and focus on how to find the coefficients that minimize the total cost.


## A worked example

We have two data points: (1, 2) and (3, 4). We want to fit a line y = a*x + b. We will compare plain linear regression, Ridge regression, and Lasso regression with lambda = 1.

Step 1: Understand the Plain Linear Regression baseline.
For the points (1, 2) and (3, 4), the line y = x + 1 passes exactly through both points.
Check: When x=1, y=1+1=2. When x=3, y=3+1=4.
The squared error is 0 because the predictions are perfect.
The coefficients are a = 1 and b = 1.

Step 2: Calculate the Ridge Cost with lambda = 1.
The Ridge cost formula is:
Cost_ridge = (Sum of squared errors) + lambda * (a^2 + b^2)

First, we need to find the coefficients that minimize this specific cost. We cannot just use the plain regression coefficients because the penalty changes the optimal balance.
Let's expand the squared error term for a general line y = ax + b on these two points:
Error at point 1: (2 - (a*1 + b))^2 = (2 - a - b)^2
Error at point 2: (4 - (a*3 + b))^2 = (4 - 3a - b)^2

Sum of squared errors = (2 - a - b)^2 + (4 - 3a - b)^2
Expanding these terms:
(2 - a - b)^2 = 4 + a^2 + b^2 - 4a - 4b + 2ab
(4 - 3a - b)^2 = 16 + 9a^2 + b^2 - 24a - 8b + 6ab

Sum = (4 + 16) + (a^2 + 9a^2) + (b^2 + b^2) + (-4a - 24a) + (-4b - 8b) + (2ab + 6ab)
Sum = 20 + 10a^2 + 2b^2 - 28a - 12b + 8ab

Now add the Ridge penalty: lambda * (a^2 + b^2). With lambda = 1, this is a^2 + b^2.
Total Ridge Cost = (20 + 10a^2 + 2b^2 - 28a - 12b + 8ab) + (a^2 + b^2)
Total Ridge Cost = 20 + 11a^2 + 3b^2 - 28a - 12b + 8ab

To find the minimum, we look for where the slope is flat. In calculus, this means taking the derivative with respect to a and b and setting them to zero.
Derivative with respect to a: 22a - 28 + 8b = 0
Derivative with respect to b: 6b - 12 + 8a = 0

Step 3: Solve the system of equations for Ridge.
From the second equation: 6b = 12 - 8a. Divide by 6:
b = 2 - (4/3)a

Substitute this expression for b into the first equation:
22a - 28 + 8(2 - (4/3)a) = 0
22a - 28 + 16 - (32/3)a = 0
Combine the 'a' terms: 22a is 66/3 a. So, (66/3)a - (32/3)a = (34/3)a.
Combine the constants: -28 + 16 = -12.
So: (34/3)a - 12 = 0
(34/3)a = 12
a = 12 * 3 / 34
a = 36 / 34 ≈ 1.0588

Now find b:
b = 2 - (4/3) * 1.0588
b = 2 - 1.4118
b ≈ 0.5882

Result for Ridge: The coefficients are a ≈ 1.06 and b ≈ 0.59.
Note that these are shrunk compared to the plain regression values of a=1, b=1. The penalty pulled them closer to zero.

Step 4: Analyze Lasso with lambda = 1.
The Lasso cost formula is:
Cost_lasso = (Sum of squared errors) + lambda * (|a| + |b|)
With lambda = 1:
Cost_lasso = (20 + 10a^2 + 2b^2 - 28a - 12b + 8ab) + |a| + |b|

Lasso is difficult to solve with simple derivatives because of the absolute value signs. However, we can test specific scenarios.
Scenario A: Assume both a and b are positive. Then |a|=a and |b|=b.
Cost = 20 + 10a^2 + 2b^2 - 28a - 12b + 8ab + a + b
Cost = 20 + 10a^2 + 2b^2 - 27a - 11b + 8ab
Taking derivatives and solving this system yields coefficients very close to the Ridge solution, but slightly smaller.

Scenario B: Assume Lasso forces one coefficient to zero. This is a key feature of Lasso. Let's test if setting b = 0 gives a lower cost than keeping b non-zero.
If b = 0, the model is y = ax. The squared error sum becomes:
(2 - a)^2 + (4 - 3a)^2 = 4 - 4a + a^2 + 16 - 24a + 9a^2 = 20 - 28a + 10a^2
The penalty term is |a| + |0| = |a|. Assuming a > 0, this is just a.
Total Cost for b=0 = 20 - 28a + 10a^2 + a = 20 - 27a + 10a^2

To minimize this quadratic cost, we find the vertex. The minimum of Ax^2 + Bx + C is at x = -B/(2A).
Here A = 10, B = -27.
a = -(-27) / (2 * 10) = 27 / 20 = 1.35

Calculate the cost at a = 1.35:
Cost = 20 - 27(1.35) + 10(1.35)^2
Cost = 20 - 36.45 + 10(1.8225)
Cost = 20 - 36.45 + 18.225
Cost = 1.775

Compare this to the plain regression cost with penalty:
Plain coefficients a=1, b=1.
Squared error = 0.
Penalty = |1| + |1| = 2.
Total Cost = 0 + 2 = 2.

Since 1.775 is less than 2, the model with b=0 and a=1.35 is better than the plain regression model under Lasso penalty.

Result for Lasso: With lambda = 1, Lasso sets b to exactly 0 and adjusts a to 1.35. This demonstrates 'sparsity': Lasso eliminated one variable entirely.


## From scratch, in Python

```python
# Plain Python implementation of ridge and lasso linear regression
# We'll train a model on a small dataset from scratch.

# Our data: house prices (in lakhs) based on size (sq ft) and number of bedrooms
data = [
    (800, 2, 30),   # size, bedrooms, price
    (1000, 3, 45),
    (1200, 3, 55),
    (1500, 4, 70),
]

# Separate inputs (list of lists) and outputs
X = [[size, bedrooms] for size, bedrooms, _ in data]
y = [price for _, _, price in data]

# Add a column of 1s for the intercept term
# We'll treat the intercept as an extra weight that multiplies 1.
X_with_bias = [[1, size, bedrooms] for size, bedrooms in X]

# Initialize weights to zeros: [intercept, slope1, slope2]
weights = [0.0, 0.0, 0.0]

# Set lambda (penalty strength) and learning rate
alpha = 0.01   # learning rate (we'll learn this later)
lambda_val = 1.0  # pick a lambda

# Training: gradient descent for a fixed number of iterations
iterations = 1000
n = len(X_with_bias)  # number of data points

for _ in range(iterations):
    # Compute predictions and errors for all points
    errors = [0.0] * n
    preds = [0.0] * n
    for i in range(n):
        row = X_with_bias[i]
        # dot product: w0*1 + w1*size + w2*bedrooms
        pred = 0.0
        for j in range(len(row)):
            pred += weights[j] * row[j]
        preds[i] = pred
        errors[i] = y[i] - pred
    
    # Update each weight
    for j in range(len(weights)):
        # The gradient of the squared error part: -2 * sum(errors * X_j)
        grad_data = 0.0
        for i in range(n):
            grad_data += -2 * errors[i] * X_with_bias[i][j]
        
        if j == 0:
            # intercept: no penalty
            grad = grad_data
        else:
            if lambda_val == 0:
                # plain regression
                grad = grad_data
            else:
                # We'll handle ridge here. Lasso is similar but with a different penalty.
                # For ridge, the gradient of lambda * w_j^2 is 2 * lambda * w_j
                grad_penalty = 2 * lambda_val * weights[j]
                grad = grad_data + grad_penalty
        
        weights[j] -= alpha * grad

# Note: This is a very simple implementation (ridge). For lasso, we'd need a different approach.
# For lasso, the penalty derivative is not smooth, so we usually use coordinate descent or 
# a subgradient method. We'll show a simple lasso later.

print("Ridge weights (b, size, bedrooms):", weights)

# Let's do a simple prediction for a 1300 sq ft, 3-bed house
new_house = [1, 1300, 3]
pred = weights[0]*new_house[0] + weights[1]*new_house[1] + weights[2]*new_house[2]
print("Predicted price (lakhs):", pred)

```


## The same thing, with a framework

```python
from sklearn.linear_model import Ridge, Lasso

# Let's use the same data
X = [[800, 2], [1000, 3], [1200, 3], [1500, 4]]
y = [30, 45, 55, 70]

# Ridge model
ridge = Ridge(alpha=1.0)  # alpha is the penalty strength (lambda)
ridge.fit(X, y)

# Predicted weights (coefficients) and intercept
print("Ridge intercept:", ridge.intercept_)
print("Ridge coefficients:", ridge.coef_)

# Lasso model
lasso = Lasso(alpha=1.0)
lasso.fit(X, y)
print("Lasso intercept:", lasso.intercept_)
print("Lasso coefficients:", lasso.coef_)

# Notice that some lasso coefficients may be exactly 0.0, especially with more features.
```


## What the code does

The from-scratch code defines a simple dataset of house prices. We add a column of 1s to the input matrix so the intercept is just another weight. We initialize weights to zero and run gradient descent, which repeatedly adjusts each weight by subtracting the gradient of the cost function. For ridge, we added the penalty term to the gradient. For lasso, the gradient is not smooth (it has a kink at zero), but a simple approach is to use a subgradient or just shrink weights by a constant amount. The framework version uses scikit-learn's built-in Ridge and Lasso classes. The library does the hard math for us, but the concept is exactly the same: it minimizes the cost with the penalty.


## Common misconceptions

**Ridge and Lasso are completely different algorithms** — Ridge and Lasso are two unrelated ways to do linear regression.

They are both just linear regression with a different penalty. The only difference is whether they add the squared sum or the absolute sum of the coefficients. In fact: They share the same core, but differ in how they shrink: ridge shrinks smoothly but never to zero, lasso can force coefficients exactly to zero.

**The intercept is penalized too** — I should penalize all the weights including the intercept.

The intercept only shifts the line up or down; penalizing it would make the model biased without reducing overfitting. In fact: The penalty applies to the coefficients of the features, not the intercept. Shifting the line doesn't create wiggles, so it's never penalized.

**Larger lambda is always better** — If a little regularization helps, then a lot must help more.

Too much penalty shrinks the coefficients too much, and the model becomes too simple (underfitting). It will no longer capture the true pattern. In fact: Lambda must be tuned. A moderate value reduces variance without introducing too much bias. You use cross-validation to pick the right value.

**Lasso is always better than ridge** — Since lasso can set coefficients to zero, it's a superior method.

When features are highly correlated, lasso will pick one at random and ignore others, making the model unstable. Ridge handles correlated features better. In fact: Ridge is better for many-item correlated features; lasso is better when you need feature selection or many irrelevant features. Elastic net combines both to get the best of both.

**Regularization only matters for big datasets** — Overfitting is only a problem when you have lots of data.

Overfitting is most severe with small datasets and many features. Regularization is especially important then. In fact: The more features you have relative to data, the more variance you have. Regularization is critical exactly when data is sparse.


## In short

Ridge and Lasso are two ways to keep a linear model simple. They add a penalty term to the cost: ridge adds the sum of squared coefficients, lasso adds the sum of absolute values. Ridge shrinks coefficients but never to zero; lasso can drive them to zero, doing feature selection. The strength of the penalty is controlled by a parameter (lambda), which you must tune. These techniques are everywhere in applied ML and even in modern deep learning as weight decay. Remember: regularization trades a little training accuracy for much better performance on new data.


## Where this is used

Ridge and Lasso are everywhere in machine learning, from simple prediction models to large language models.

**In large language models (LLMs):** The idea of adding a penalty to weights appears as weight decay, which is the same as L2 regularization (ridge). Modern LLM training uses weight decay to keep the model from overfitting to the training data. It's not called ridge, but the math is identical. For example, in the AdamW optimizer, weight decay is applied directly to the weights, not through the gradient. This helps LLMs generalize to new text.

**In small models on devices:** When you run a model on a phone or a laptop, you care about size and speed. Lasso is useful here because it can set many coefficients to zero, meaning those features are not used at all. This *prunes* the model, making it smaller and faster. A practical example: a keyword-spotting model that only needs a few important audio features; lasso will zero out the irrelevant ones, so the model runs efficiently on a small chip.

**In retrieval-augmented generation (RAG):** RAG retrieves documents relevant to a query and feeds them into an LLM. Regularization isn't directly used in the retrieval step, but when you train a re-ranker to score documents, ridge or lasso can help it generalize to new queries. Also, when you embed documents into vectors, you might use a linear model on those vectors; ridge can prevent overfitting when the dataset is small.

**In agentic systems that plan and call tools:** Agents often need to decide which tool to call. If they have a model that scores actions based on features (like similarity, reward estimate), a regularized linear model can learn which features matter without overfitting. Lasso's feature selection is useful when there are many possible sensor inputs or API parameters.

**In everyday applied work:** 
- *Recommendation systems:* Collaborative filtering with many users and items. Ridge prevents overfitting when predicting user ratings.
- *Credit scoring:* Banks use logistic regression (which can be regularized) to predict default. Lasso helps identify which financial factors are most predictive.
- *Forecasting:* Time series models often use regularization to handle many lags.
- *Fraud detection:* With thousands of transaction features, lasso selects the few that signal fraud, making the model interpretable.
- *Clinical prediction:* In medical data with many biomarkers, ridge or lasso are the standard to avoid overfitting to noise.

**What developers actually do:** A developer working on a house-price predictor would use scikit-learn's `Ridge` or `Lasso` classes, usually in a pipeline with standardization (scaling features) and cross-validation to tune lambda. They write:
```python
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Lasso

model = make_pipeline(StandardScaler(), Lasso(alpha=0.1))
model.fit(X_train, y_train)
```
They then evaluate on a holdout set. If they use lasso, they can look at `model.named_steps['lasso'].coef_` and see exactly which features have non-zero weights — a sparsity pattern that helps interpretability.

**What breaks when done badly:** If you use ridge with a lambda too large, the model becomes too simple and underfits — it predicts almost nothing. If you use lasso and have highly correlated features, it will pick one arbitrarily and completely ignore the other, which can be unstable. That's why elastic net (combining both) exists. Also, if you forget to standardize your features, the penalty will apply unevenly (a feature with large values will get more penalty), leading to wrong results.


---

Read this on the web: https://basavapurushottam.com/learn-ai/ridge-and-lasso-regression
