# Partial Least Squares

A regression method that constructs components maximising covariance with the response, rather than variance in the predictors alone as PCA does.

**Rests on:** Normal Equations

*Mathematical Foundations · Statistical Inference*


## Why learn this

This helps you build reliable predictions when you have many inputs that are noisy or overlap with each other. Imagine you want to predict crop yield from 200 soil measurements, or forecast sales from 30 different marketing channels. In these cases, ordinary prediction methods fail because the data is messy and contradictory. Partial Least Squares solves this by finding fewer, cleaner summary numbers that are specifically useful for predicting your target. You will learn how to choose the right number of these summaries so your model works well on new data, rather than just memorizing old mistakes.


## The intuition

Imagine you have a big table of numbers — say, measurements from 50 sensors on a machine, and you want to predict how many hours the machine will run before it fails. Each sensor is a 'predictor', and the failure time is your 'response'. If you tried to use all 50 sensors in a plain regression, the model would be a mess: many sensors are measuring the same underlying thing (temperature, vibration, pressure), so they are collinear — meaning their values rise and fall in lockstep, making it impossible for the model to distinguish which specific sensor is actually driving the change. The model can't tell which one matters, and its predictions become unstable: a tiny change in the data flips the answer. What you really want is to find the few underlying 'forces' — maybe one is 'overall heat stress', another is 'mechanical wear' — that drive both the sensors and the failure time. Partial Least Squares (PLS) does exactly that. It looks at the sensors and the failure times together, and pulls out a small number of 'components' — each one a weighted combination of the sensors — that explain both the variation in the sensors and, crucially, the variation in failure time. It's like a detective who, instead of interviewing every witness separately (and getting confused by contradictions), groups the witnesses by what they actually saw, and uses each group's consensus to solve the crime. The key difference from a method called PCA (Principal Component Analysis) is that PCA only looks at the sensors — it finds directions of high variance in the sensors alone, without caring about the failure time. PLS is 'supervised': it lets the response guide which components to build, so the components are the ones that actually predict well. In practice, you end up with a model that uses just a handful of components instead of 50 predictors, is more stable, and often predicts better because it ignores the noise.


## An analogy

Think of a group of tourists trying to find the best restaurant in a city. They have a map with many landmarks (predictors), but they don't know which landmarks matter. PLS is like a savvy local guide. The guide doesn't just look at the map and pick the most prominent landmarks (that would be PCA — it might pick a big shopping mall that has nothing to do with food). Instead, the guide watches where the locals actually go (the response). The guide notices that whenever locals go to a certain neighborhood, they end up at good restaurants. So the guide combines several landmarks — the presence of a park, a metro station, and a bakery — into a single 'restaurant-friendliness' score (a component). This score is built specifically to predict where the locals go. The guide repeats this a few times, each time finding a new combination of landmarks that explains the remaining 'food-seeking' behavior. At the end, the guide has a handful of scores — not 50 separate landmarks — that predict restaurant quality well. Where the analogy breaks down: the guide is a human with intuition; PLS is a mathematical algorithm that finds these combinations automatically by maximizing a specific numerical goal (covariance, which measures how much two things vary together). Also, the guide can see all the landmarks at once; PLS builds components one at a time, each new one accounting for what the previous ones missed. And crucially, the guide's judgment is subjective; PLS's components are deterministic — given the same data, you always get the same components.


## Definition

Partial Least Squares (PLS) is a prediction method that creates fewer summary numbers from many inputs by finding combinations of those inputs that vary together with the outcome you want to predict. It projects the predictors and the response into a lower-dimensional space by constructing components that maximize the covariance between them, and then fits a linear regression on those components.


## How it connects

You haven't met PCA yet, but you have met the idea of correlation and scatterplots — how two variables move together. PLS builds on that by looking at the joint variation between many predictors and one response. It's related to ANOVA in the sense of partitioning variance, but here we partition the variance of predictors *and* response together. And it connects directly to Cross-Validation — you'll use cross-validation to choose how many components to keep, because using too many overfits and too few underfits. Think of PLS as the supervised cousin of PCA, which you'll learn later; both reduce dimensions, but PLS uses the response to guide the reduction.


## The mathematics

The mathematics behind PLS rests on **linear algebra** — the branch of maths that deals with lists of numbers and tables of numbers, and how to combine them. You don't need to have studied it before; we'll build every idea from scratch. The key operation is the **dot product**, which measures how much two lists of numbers agree with each other. If you have two lists, say (1, 2, 3) and (4, 5, 6), the dot product is: multiply the first numbers together, then the second numbers, then the third, and add all those products: (1×4) + (2×5) + (3×6) = 4 + 10 + 18 = 32. That single number tells you how aligned the lists are — if they were both pointing in the same direction, the dot product would be large and positive; if they pointed in opposite directions, it would be negative. Now, in PLS we have a table of predictors (rows are samples, columns are sensors) and a column of responses. We want to find a new 'component' — a weighted combination of the predictors — that has a high dot product with the response. That is, we want the component to 'agree' with the response. The dot product between the component scores and the response is called the **covariance** (roughly). PLS finds the weights that make this covariance as large as possible. It does this repeatedly: after finding one component, it removes the variation in the predictors and the response that this component explains, and then finds the next component that maximises covariance with the *remaining* response. After a few components, you have a small set of scores that capture the predictive information. Then you fit a simple regression: the response ≈ a constant plus (weight1 × component1) plus (weight2 × component2), and so on. To make a prediction for a new sample, you compute its component scores using the same weights, and plug them into that equation. That's the whole idea in words. Now let's do it with real numbers.

Let X be the matrix of centred predictors (rows = samples, columns = predictors), and y be the centred response vector. PLS finds weights w (a vector) that maximise:

max ||w||=1, ||t||=1 Cov(Xw, y) = max (t^T X w)^T (t^T y)

The first component score is t1 = X w1, where w1 is the first weight vector. Then we subtract the variation explained: X_new = X - t1 p1^T, y_new = y - t1 c1, where p1 and c1 are regression coefficients of X and y on t1. Repeat. After K components, the prediction is y_new = sum_{k=1}^K t_k c_k.

Symbol glossary:
- X: the matrix of predictor values, with rows as samples and columns as predictors
- y: the vector of response values
- t_k: the k-th component score (a vector of scores for all samples)
- w_k: the k-th weight vector, which defines the component as a weighted sum of predictors
- p_k: the loading vector for the predictors in the k-th component
- c_k: the regression coefficient of the response on the k-th component
- K: the number of components chosen, which is the key tuning parameter
- Cov: covariance, a measure of how two sets of values move together
- ||.||: the length (norm) of a vector, used to normalise the weights


## A worked example

```
We have 3 samples. Each sample has 2 predictors (x1, x2) and one response (y). Data: Sample A: x1=1, x2=3, y=5. Sample B: x1=2, x2=4, y=7. Sample C: x1=3, x2=6, y=11. We want to build a PLS model with 1 component.

Step 1: First, centre the data (subtract the mean of each column). Mean of x1 = (1+2+3)/3 = 2. x1 centred: -1, 0, 1. Mean of x2 = (3+4+6)/3 = 4.333; x2 centred: -1.333, -0.333, 1.667. Mean of y = (5+7+11)/3 = 7.667; y centred: -2.667, -0.667, 3.333.   # Centring makes the data have mean zero, so the components are about variation, not absolute levels.
Step 2: Step 2: Find the weights that maximise the covariance between the component score and the response. This involves solving a small matrix problem. For our data, the solution is approximately weights w1=0.447, w2=0.894 (these are normalised so that the component has unit length).   # These weights tell us how much each predictor contributes to the component. Here, x2 gets about twice the weight of x1, matching the fact that y increases more with x2.
Step 3: Step 3: Compute the component score for each sample. Score = w1*x1_centred + w2*x2_centred. For sample A: 0.447*(-1) + 0.894*(-1.333) = -0.447 - 1.192 = -1.639. For sample B: 0.447*0 + 0.894*(-0.333) = -0.298. For sample C: 0.447*1 + 0.894*1.667 = 0.447 + 1.490 = 1.937.   # These scores are a single number per sample — a 'compressed' representation of the two predictors.
Step 4: Step 4: Fit a simple regression of the centred response on the component scores. The best-fit line is approximately y_centred = 2.1 * score.   # This is a one-variable regression — easy to fit — instead of a two-variable one.
Step 5: Step 5: Check predictions. For sample A, predicted y_centred = 2.1 * (-1.639) = -3.44, so predicted y = -3.44 + 7.667 = 4.23 (actual 5). For sample B: 2.1 * (-0.298) = -0.63, predicted y = 7.04 (actual 7). For sample C: 2.1 * 1.937 = 4.07, predicted y = 11.74 (actual 11).   # The model captures most of the variation in y with just one component.

Result: With just one component, we get reasonable predictions (errors around 0.5–0.8). With two components (the maximum possible), we would get a perfect fit, but that would be overfitting — the goal is to use the fewest components that predict well on new data.
```


## From scratch, in Python

```python
import numpy as np

# Data: 3 samples, 2 predictors, 1 response
X = np.array([[1.0, 3.0],
              [2.0, 4.0],
              [3.0, 6.0]])
y = np.array([5.0, 7.0, 11.0])

# Centre the data
X_centered = X - X.mean(axis=0)
y_centered = y - y.mean()

# Number of components to extract
n_components = 1

# Storage for weights, scores, loadings, coefficients
W = []
T = []
P = []
C = []

# Copy the centred matrices so we can alter them
X_current = X_centered.copy()
y_current = y_centered.copy()

for k in range(n_components):
    # Compute the weight vector: the direction in X that maximises covariance with y
    # w = X^T y / ||X^T y||
    w = X_current.T @ y_current
    w = w / np.linalg.norm(w)
    W.append(w)

    # Compute the scores: t = X w
    t = X_current @ w
    T.append(t)

    # Compute the loading for X: p = X^T t / (t^T t)
    p = X_current.T @ t / (t @ t)
    P.append(p)

    # Compute the regression coefficient for y: c = y^T t / (t^T t)
    c = (y_current @ t) / (t @ t)
    C.append(c)

    # Deflate: remove the variation explained by this component
    X_current = X_current - np.outer(t, p)
    y_current = y_current - t * c

# Convert lists to arrays for convenience
W = np.array(W)
T = np.array(T)
P = np.array(P)
C = np.array(C)

# Predict for new data (e.g., original X)
# Compute scores for each component and sum contributions
def predict(X_new, W, P, C, X_mean, y_mean):
    X_new_centered = X_new - X_mean
    scores = X_new_centered @ W.T  # shape (n_samples, n_components)
    y_pred_centered = scores @ C    # C is a vector of length n_components
    return y_pred_centered + y_mean

# Test on the training data
y_pred = predict(X, W, P, C, X.mean(axis=0), y.mean())
print("Predictions:", y_pred)
print("True y:", y)
print("Weights:", W)
print("Scores:", T)
```


## The same thing, with a framework

```python
from sklearn.cross_decomposition import PLSRegression
import numpy as np

X = np.array([[1.0, 3.0],
              [2.0, 4.0],
              [3.0, 6.0]])
y = np.array([5.0, 7.0, 11.0])

pls = PLSRegression(n_components=1)
pls.fit(X, y)

y_pred = pls.predict(X)
print(y_pred)
# This is essentially the same algorithm we implemented above. The library handles centring, deflation, and prediction efficiently, but underneath it's computing the same weights and scores.
```


## What the code does

We start by importing numpy and defining our data. The key idea is to centre the data — subtract the mean of each column — so that we're working with variation, not levels. Then we loop over the number of components. For each component, we compute the weight vector w as the direction in the predictor space that maximises covariance with the response. Mathematically, that's the vector X^T y (the dot product of each predictor with the response), normalised to unit length. This is the 'supervised' step — without y, this would be PCA. Then we compute the scores t as the projection of X onto w. We then find the least-squares coefficients p and c that best explain X and y from t. Finally, we 'deflate': we subtract the variation already explained, so the next component finds something new. The prediction function re-centres new data using the training means (important — never use test data to centre), computes the scores, and linearly combines them with c, then adds back the response mean. The framework version using scikit-learn's PLSRegression does exactly this under the hood — you don't need to write the loops, but now you know what it's doing.


## Common misconceptions

**PLS is the same as PCA** — PLS is just PCA performed on both X and y at the same time.

PCA finds components that maximise variance in X alone, ignoring y. PLS uses y to guide the components, maximising covariance with y. That makes PLS supervised, which is the entire point: the components are predictive, not just descriptive. In fact: PLS is supervised: it constructs components that maximise the covariance between the predictor scores and the response. PCA is unsupervised: it only looks at the predictors.

**More components always means better predictions** — If I use more components, the model fits the data better, so predictions on new data will be better too.

Adding components reduces training error but can severely overfit, capturing noise that doesn't generalise. Cross-validation shows that test error typically goes down then up as you add components. In fact: The number of components is a bias-variance tradeoff. You choose it by cross-validation — the number that minimises validation error, not training error.

**PLS works even if predictors are on wildly different scales** — The algorithm automatically handles scale differences, so I don't need to normalise.

PLS maximises covariance, and covariance is scale-sensitive. A predictor measured in meters will dominate one measured in micrometers simply because the numbers are larger, regardless of its actual predictive power. In fact: Always standardise predictors (subtract mean, divide by standard deviation) before running PLS, especially when predictors are on different units or scales.

**PLS can handle any type of relationship (nonlinear)** — PLS is powerful enough to capture nonlinear relationships between predictors and response.

PLS is a linear method. It creates linear combinations of predictors and fits a linear regression in the component space. If the true relationship is curved, PLS will miss it. In fact: If you have nonlinear relationships, use kernel PLS, or pre-process the predictors with nonlinear transformations (e.g., logs, polynomials) before applying PLS.

**PLS coefficients are the same as ordinary regression coefficients** — The component weights tell me the direct effect of each predictor on the response, just like in linear regression.

PLS weights define the components, but the relationship between the original predictors and the response is a sum of contributions through all components. A predictor can have a positive weight in one component and negative in another, making its net effect unclear. In fact: To interpret the effect of an original predictor, you must combine the component weights and regression coefficients — the 'model coefficients' in the original space are the sums over components of w_k * c_k. They are not the same as individual component weights.


## In short

Partial Least Squares is a supervised method for regression when you have many collinear predictors. It finds a few components — weighted combinations of predictors — that maximise covariance with the response, then regresses the response on those components. The number of components is the key tuning choice, selected by cross-validation. It is a mainstay of applied machine learning, especially in spectroscopy, chemometrics, and any domain where predictors outnumber samples, and it is easy to implement and interpret in scikit-learn.


## Where this is used

Partial Least Squares is a workhorse in chemometrics, spectroscopy, and many industrial and biological settings, but it also appears in modern AI systems in specific, concrete ways that you should know about.

**In everyday applied machine learning** — the most common use. Imagine you're a data scientist at a food company, and you have near-infrared (NIR) spectra of 1000 samples of flour. Each spectrum has 500 wavelengths (predictors), and you want to predict the protein content (response). The wavelengths are extremely collinear — neighbouring wavelengths are almost identical. Ordinary regression will give you coefficients that bounce around wildly; one day the model says wavelength 100 has a huge positive effect, the next day it's negative. PLS solves this by compressing the 500 wavelengths into, say, 10 components that predict protein content well. The developer writes: from sklearn.cross_decomposition import PLSRegression; pls = PLSRegression(n_components=10); pls.fit(X_spectra, y_protein); and then pls.predict(new_spectra). The critical decision is choosing n_components — too few and you underfit (you miss real structure), too many and you overfit (you model noise). You choose it by cross-validation: split the data into folds, fit PLS with each candidate number of components on the training folds, and measure prediction error on the validation fold. The number with the lowest cross-validated error wins. The same workflow applies to any domain where predictors outnumber samples and are collinear: genetics (thousands of gene expression values), economics (many correlated indicators), sensory analysis, and so on.

**In small models running on devices** — PLS is not usually the first choice for deep learning, but for extreme edge situations it shines. When you have to deploy a model on a microcontroller with, say, 2KB of RAM, you can't run a neural network with millions of parameters. But you can run a PLS model with 5 components: the prediction is just a handful of dot products. In fact, PLS is often used in portable spectrometers — handheld devices that shine light on materials and identify their composition. The device captures a spectrum, computes a few component scores, and outputs a prediction — all in less than a millisecond. The developer would implement the prediction as a fixed sequence of multiply-accumulate operations, perhaps in C, using the weights and coefficients extracted from the training run. The model is small, fast, and requires no floating-point heavy libraries.

**In large language models (LLMs)** — PLS itself is not used inside LLMs. LLMs are trained with gradient descent on huge neural networks, and their components are learned hidden states, not covariance-maximising projections. However, the *principle* of PLS — finding low-dimensional structure that predicts a target — appears in a related technique called 'probing'. Researchers train small linear models (sometimes using PLS or PCA on the hidden states) to see if a certain property (like sentiment or part-of-speech) is linearly detectable in the representations. This is a diagnostic tool, not part of the model's inference. So if you hear 'PLS and LLMs', it's usually about analysing the model, not building it.

**In retrieval-augmented generation (RAG)** — RAG relies on embeddings: turning text into vectors, then finding similar vectors by cosine similarity. PLS is not typically used to build embeddings (that's the job of neural encoders), but it can be used as a dimensionality reduction step after obtaining embeddings. If you have thousands of embedding dimensions and you want to speed up search, you might reduce the dimension with PLS, where the 'response' could be a label like relevance. This is a niche use, though; PCA is more common for pure dimensionality reduction because it doesn't need the label. PLS would be used if you specifically want the reduced dimensions to be predictive of some target, e.g., to compress the embeddings while retaining the information that predicts relevance.

**In agentic systems that plan and call tools** — Again, PLS is not a core component. Agents use LLMs, planning algorithms, and tool calls. However, PLS can appear in the *environment* that the agent interacts with. For example, an agent that controls a manufacturing process might use a PLS model as a sensor fusion module: it takes readings from multiple sensors (collinear) and outputs a simplified state estimate that the agent then uses for planning. The agent doesn't need to know PLS; it just sees the reduced state. So PLS serves as a pre-processing module, not part of the agent's reasoning loop.

**When building software with PLS** — A developer reaches for scikit-learn (or MATLAB, since your library mentions the MATLAB Toolbox). In scikit-learn, the code is: PLSRegression(n_components=k).fit(X, y). The main loop of model development is: 1) preprocess (centre, scale, maybe log-transform), 2) use cross-validation to choose k, 3) fit the final model with that k, 4) inspect the weights and loadings to interpret which original predictors matter (this is a major advantage of PLS — you can see which wavelengths or genes drive the prediction), 5) deploy. What breaks when done badly: choosing k by minimising training error (you'll overfit), not scaling the predictors when they're on very different scales, and leaking information by centring using the full dataset before splitting — that leaks information because the means include the validation data. You must centre per-fold inside cross-validation. Also, the interpretation of weights can be misleading if you don't check the model's assumptions, such as linearity — PLS assumes a linear relationship between components and response.

**In forecasting, fraud detection, and vision** — PLS is used in forecasting when you have many correlated time series as predictors. In fraud detection, if you have a large set of transaction features, PLS can compress them into a few scores that predict fraud status. In computer vision, PLS has been used for face recognition (e.g., the 'eigenfaces' variants) where it projects face images onto a few components that are correlated with identity labels. Again, the mechanism is the same: find components that predict the target.

So PLS is a widely useful tool for applied statistical modelling, especially where predictors are collinear and numerous. It is not a part of modern deep learning architectures, but it remains the standard approach in many scientific and industrial settings, and you'll often see it in conjunction with neural models as a feature extraction or stabilisation step.


---

Read this on the web: https://basavapurushottam.com/learn-ai/partial-least-squares
