# Dimensionality Reduction and PCA

Projecting data onto the directions of greatest variance, compressing many correlated features into a few components.

**Rests on:** Least Squares Regression, Tensors and Shapes

**Leads to:** K-Means Clustering

*Machine Learning · Unsupervised Learning*


## Why learn this

Imagine you have a spreadsheet describing 100 different characteristics of 10,000 customers—age, income, purchase history, website clicks, and so on. Trying to find patterns in such a wide table is like trying to listen to a conversation in a noisy stadium. Dimensionality reduction lets you focus on the few most important signals instead of getting lost in the noise. Specifically, this technique helps you shrink complex data down to its essential shape, making it easier to visualize, faster for computers to process, and clearer for spotting unusual cases. After studying this, you will be able to take a messy, wide dataset, identify the key trends that define it, and summarize that dataset using just two or three numbers without losing the main story the data is telling.


## The intuition

Picture yourself in a crowded room where everyone is described by 100 different metrics: height, weight, shoe size, income, hours slept, coffee consumed, etc. You want to understand if there are distinct groups of people, but staring at all 100 numbers is overwhelming. Dimensionality reduction is like asking: 'If I had to describe everyone in this room using only two sentences, what would those two sentences be?' PCA (Principal Component Analysis) does exactly this by finding the most important directions in which the data varies.

Let’s look at a smaller example. Imagine you are describing books in a library using three attributes: number of words, number of images, and density of technical jargon. You notice that books with many words also tend to have few images (they are text-heavy) and high jargon (they are scientific). The data points on your chart form a long, thin cigar shape stretching diagonally. Instead of keeping all three axes, you can rotate your view. You create a new 'axis' that runs along the length of the cigar. This single new axis captures almost all the variation in the data. The width of the cigar is negligible. So, you can describe every book using just one number: its position along this new 'complexity axis'. This new axis is called a principal component. It is not one of your original measurements; it is a calculated blend of them that points in the direction where the data spreads out the most.

How are these directions found? You start by calculating how much each attribute varies on its own (its variance). Then you look at how attributes change together (covariance). The principal components are specific mathematical directions that maximize this spread. The first component is the direction with the greatest spread. The second component is the next most important direction, but it must be perpendicular to the first. It captures the remaining major patterns. By keeping only these top directions, you compress your data. You lose some fine detail (like exactly how many images a specific book has), but you keep the essential structure (that it is a scientific text).


## An analogy

Think of standing around a large, irregular rock sculpture in a park. If you take a photograph from one angle, the shadow on the ground might look like a triangle. From another angle, it might look like an oval. A 'scree plot' is like a checklist where you measure how much detail (variation) each different shadow captures and rank them from most to least. PCA finds the single best angle (or set of angles) such that when you project the 3D rock onto a 2D plane, the resulting shape retains as much of the original object's distinguishing features as possible. The principal components are essentially those optimal camera angles. Caveat: While shadows can look very different depending on the light source, PCA always finds the mathematically 'best' view for preserving statistical spread, regardless of what the object looks like from a casual glance.


## Definition

Principal Component Analysis is a method for simplifying data by creating new, combined features called principal components that point in the directions where the data varies the most, allowing you to represent complex information using fewer numbers.


## How it connects

This technique sits within unsupervised learning, which focuses on finding hidden patterns in data without labeled answers. It shares space with clustering methods like k-means (which groups similar items), DBSCAN (which finds clusters of any shape based on density), and hierarchical clustering (which builds a tree of nested groups). While clustering tries to separate data into buckets, PCA tries to condense the data itself. It also connects to embeddings, which are complex numerical representations used by AI models to understand words or images, as both techniques aim to capture meaningful structure in high-dimensional space.


## The mathematics

This concept rests on linear algebra—the mathematics of lists of numbers and the operations you can do with them. You will need three operations: the mean (average), variance (spread), and the dot product (a measure of how much two lists point in the same direction). PCA works in steps: first, you centre the data by subtracting the mean of each feature—this moves the cloud so its average is at the origin. Second, you measure how each feature varies alone (its variance) and how it varies together with each other feature (covariance). These measurements fill a table called the covariance matrix. Third, you find the directions in which the data has the most variance—these are the eigenvectors of that matrix, and the amount of variance each one captures is the eigenvalue. Finally, you order these directions by their eigenvalues and keep the top few, projecting your data onto them. That projection is your compressed representation.

PCA transformation: Z = X_centre @ W, where X_centre is the centred data matrix, W is the matrix of top k eigenvectors (as columns), and @ is matrix multiplication. The covariance matrix is C = (1/(n-1)) * X_centre^T @ X_centre. Eigenvectors of C are found by solving Cv = λv. The first principal component is the eigenvector with the largest eigenvalue λ.


## A worked example

```
You have 3 data points, each with 2 features (x, y): (1, 2), (3, 4), (5, 6). We will compute the covariance matrix and find the principal component by hand.

Step 1: Mean of x = (1+3+5)/3 = 3   # The average of the first feature.
Step 2: Mean of y = (2+4+6)/3 = 4   # The average of the second feature.
Step 3: Centre the data: (1-3, 2-4) = (-2, -2); (3-3, 4-4) = (0, 0); (5-3, 6-4) = (2, 2)   # Subtract the means so the cloud is centred at (0,0).
Step 4: Covariance (x,x) = [(-2·-2) + (0·0) + (2·2)] / (3-1) = (4+0+4)/2 = 4   # This is the variance of x, the average squared deviation from the mean, with Bessel's correction (divide by n-1).
Step 5: Covariance (x,y) = [(-2·-2) + (0·0) + (2·2)] / (3-1) = 4   # Since x and y move together here, the covariance is positive and equals the variance.
Step 6: Covariance (y,y) = [(-2·-2) + (0·0) + (2·2)] / (3-1) = 4   # The data is symmetric in x and y.
Step 7: Covariance matrix = [[4, 4], [4, 4]]   # Arrange the covariances in a square table.
Step 8: Find eigenvectors: for this matrix, the eigenvectors are approximately (0.707, 0.707) with eigenvalue 8, and (-0.707, 0.707) with eigenvalue 0.   # The directions of greatest and least variance; the first captures all the spread.
Step 9: Project data onto first eigenvector: (-2, -2)·(0.707, 0.707) = -2.828; (0,0)·(0.707, 0.707) = 0; (2,2)·(0.707, 0.707) = 2.828   # Each centred point is replaced by a single number—the dot product with the principal direction.

Result: The 3 two-dimensional points are compressed to 3 one-dimensional values: -2.828, 0, 2.828. You have reduced dimensionality from 2 to 1 while preserving the variance (spread) of the data.
```


## From scratch, in Python

```python
import math

# Our data: a list of lists, each inner list is one data point (x, y)
data = [[1, 2], [3, 4], [5, 6]]

# Step 1: Compute the mean of each feature
# We need to loop over all data points, gather the x-values, and average them.
# We can use a list comprehension: [row[0] for row in data] gives [1, 3, 5]
mean_x = sum([row[0] for row in data]) / len(data)
mean_y = sum([row[1] for row in data]) / len(data)

# Step 2: Centre the data
centred = []
for row in data:
    centred.append([row[0] - mean_x, row[1] - mean_y])

# Step 3: Compute covariance matrix (2x2)
# n is the number of data points, n-1 for sample covariance
n = len(data)
# Use list comprehensions again to extract columns
x_col = [row[0] for row in centred]
y_col = [row[1] for row in centred]

cov_xx = sum([xi * xi for xi in x_col]) / (n - 1)
cov_yy = sum([yi * yi for yi in y_col]) / (n - 1)
cov_xy = sum([x_col[i] * y_col[i] for i in range(n)]) / (n - 1)

# Step 4: Find the eigenvector for the largest eigenvalue
# For a 2x2 matrix [[a, b], [c, d]], the eigenvalues solve det(C - λI) = 0
# Here a = cov_xx, b = c = cov_xy, d = cov_yy
a = cov_xx
b = cov_xy
c = cov_xy
d = cov_yy
# Characteristic polynomial: λ^2 - (a+d)λ + (ad - bc) = 0
# Solve with the quadratic formula
trace = a + d
det = a*d - b*c
# For this simple example, one eigenvalue is 0, so the largest is trace
eigenvalue = trace  # works when det = 0, as in our perfectly correlated case
# Eigenvector for [[a,b],[c,d]] corresponding to eigenvalue λ: v = (b, λ - a) or (λ - d, c)
if b != 0:
    v1 = b
    v2 = eigenvalue - a
else:
    v1 = eigenvalue - d
    v2 = c
# Normalise the vector (make its length 1)
length = math.sqrt(v1*v1 + v2*v2)
v1 /= length
v2 /= length

# Step 5: Project the centred data onto the principal component
projected = []
for row in centred:
    # dot product: row[0]*v1 + row[1]*v2
    val = row[0]*v1 + row[1]*v2
    projected.append(val)

print("Projected data:", projected)
```


## The same thing, with a framework

```python
import numpy as np
from sklearn.decomposition import PCA

# Data as a numpy array
data = np.array([[1, 2], [3, 4], [5, 6]])

# Create a PCA object that reduces to 1 dimension
pca = PCA(n_components=1)
# Fit the model and transform the data in one call
reduced = pca.fit_transform(data)

print(reduced)
# Output: [[-2.82842712]
#          [ 0.        ]
#          [ 2.82842712]]

# The principal component direction (eigenvector)
print(pca.components_)
# Output: [[0.70710678 0.70710678]]
```


## What the code does

The code builds up PCA step by step. First, it calculates the mean of each feature by taking a list comprehension—`[row[0] for row in data]` extracts the first number of every row, and `sum` adds them all, then `len(data)` counts the rows. Then it centres the data with a for loop that subtracts the mean from each coordinate. The covariance matrix is computed by again using list comprehensions to pull out the x- and y-columns, then multiplying corresponding entries and summing them. The eigenvector calculation is simplified because the example data is perfectly correlated—the covariance matrix is singular, and the largest eigenvalue is just the trace. The code normalises the eigenvector so its length is 1, which is what makes it a direction. Finally, it projects each centred point onto that direction by taking the dot product (multiply matching entries and add). If you run this, you get the three projected values you computed by hand.


## Common misconceptions

**PCA is a supervised method** — PCA needs a target variable to know which dimensions are important.

PCA is unsupervised; it looks only at the variance of the features, not at any labels. It does not consider what you are trying to predict. In fact: PCA finds directions of maximum variance in the feature space alone. It can be used before a supervised model, but it does not itself use the target.

**Principal components are just subsets of original features** — PCA picks the most important original columns, so I can interpret the components as feature names.

Each principal component is a weighted sum of all original features, not a selection. The weights can be positive or negative, so a component may combine income and debt in a way that has no simple label. In fact: A principal component is a linear combination—a blend—of all original features. This is the cost of interpretability: components are often hard to name.

**Scaling is optional** — PCA will work fine on raw features; the scale of each feature doesn't matter.

Features with larger ranges (e.g., income in lakhs vs. age) will dominate the variance calculation, and the first principal component will simply align with that high-variance feature, ignoring the others. In fact: You must standardise each feature to zero mean and unit variance (or at least equalise scales) before PCA, otherwise the result is distorted.

**PCA always improves accuracy** — Using PCA to reduce dimensions will make my model more accurate, because simpler is better.

PCA discards some information. If the discarded dimensions contain signal relevant to the target, accuracy can drop. Dimensionality reduction is a trade-off between simplicity and information loss. In fact: PCA helps with speed, storage, and visualisation, and can reduce overfitting, but it does not guarantee better predictive performance. Evaluate with cross-validation.

**PCA is the same as feature selection** — PCA and selecting a subset of features are the same thing—both reduce the number of columns.

Feature selection keeps original columns and drops others; PCA creates entirely new columns that are combinations of the old ones. The two have different effects on interpretability and downstream models. In fact: Feature selection preserves interpretability (you keep original features); PCA creates new ones, sacrificing interpretability for maximum variance preservation.


## In short

PCA is a way to compress data by finding the directions of greatest variance—the principal components. You centre the data, compute the covariance matrix, find its eigenvectors, and project onto the top few. Remember: scale your data first, components are blends not subsets, and PCA is unsupervised. This is your first step toward understanding embeddings and the internal representations of neural networks.


## Where this is used

PCA is everywhere in applied machine learning. 

**Large language models (LLMs):** Inside a transformer, the embedding layers map words to vectors of thousands of numbers. PCA is often used to visualise these high-dimensional embeddings—for example, projecting them to 2D with PCA to see clusters of related words. More fundamentally, the attention mechanism itself can be seen as learning a linear projection of keys, queries and values—a close cousin of PCA's projection, though learned rather than fixed. Developers use PCA to analyse the internal representations, to see what dimensions capture semantic concepts (like gender or tense). 

**Small models on devices:** When a model must run on a phone or a microcontroller, every kilobyte matters. PCA is used to compress feature vectors before feeding them into a small classifier—for instance, compressing 100 audio features to 20 principal components. This reduces the model's input size, making it faster and cheaper to run, while retaining most of the signal. It also helps with overfitting: fewer dimensions mean fewer parameters to learn, which is crucial when training data is scarce. 

**Retrieval-augmented generation (RAG):** RAG systems retrieve documents by comparing embedding vectors—often thousands of dimensions. While exact nearest-neighbour search is possible, PCA is used to compress these embeddings to a fraction of their size—say from 1536 to 256 dimensions—so that the vector database is smaller, retrieval is faster, and the semantic meaning is largely preserved. Some systems also apply PCA to the query embedding on the fly to match the compressed index. The trade-off is a slight loss of retrieval quality, but the speed gain is often worth it. 

**Agentic systems:** Agents that plan and use tools often have to manage long conversations. PCA is used in some agent memory systems to compress past observations into a low-dimensional state vector, so the agent can reason about the gist of a long history without processing every token. It also appears in the tool-selection stage: the agent may embed available tool descriptions and use PCA to find the most relevant ones in a compressed space. 

**Everyday applied work:** In recommendation systems, user preferences are represented as sparse vectors; PCA reduces them to dense 'latent factors' that are faster to compare. In computer vision, PCA is applied to image patches (e.g., for face recognition—eigenfaces). In fraud detection, PCA is used to reduce thousands of transaction features to a few components that are fed into an anomaly detector. In forecasting, PCA is used to compress many correlated time series into a few principal components that are then forecast, before transforming back. 

**What a developer does:** The typical workflow is: (1) Standardise the data—scale each feature to zero mean and unit variance, because PCA is sensitive to feature scales (a feature measured in rupees vs. rupees thousands will dominate). (2) Fit PCA on the training data only, and transform both training and test data. (3) Choose the number of components by looking at the explained variance ratio—often keeping 95% of the variance. In Python, this is `from sklearn.preprocessing import StandardScaler` then `from sklearn.decomposition import PCA; pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95))`. A common mistake is to fit PCA on the whole dataset before splitting, which leaks information and overstates performance—the correct order is to fit only on the training fold, then transform the test fold. 

**When done badly:** If you forget to scale, a feature with a large range (e.g., income in rupees) will dominate the first principal component, and your reduction will be meaningless—you will have thrown away the structure you wanted to keep. If you keep too few components, you lose important signal and your downstream model underperforms. If you apply PCA before splitting, you introduce leakage and your validation results become optimistic.


---

Read this on the web: https://basavapurushottam.com/learn-ai/dimensionality-reduction-and-pca
