# Eigenvalues and Eigenvectors

Directions a transformation merely stretches rather than rotates, and the factors by which it stretches them.

**Rests on:** Linear Transformations, Matrix Multiplication, Vector Spaces and Rank

*Mathematical Foundations · Linear Algebra*


## Why learn this

This concept unlocks how machines find hidden structure in data. You will use it to understand Principal Component Analysis — the technique that reduces thousands of measurements to a few meaningful ones — and to see how search engines rank pages, how recommendation systems find patterns, and why some models can be compressed to run on a phone. Without eigenvalues and eigenvectors, you cannot truly understand why PCA works, why PageRank orders results, or why certain AI systems can be shrunk without losing much. This is the gateway to multivariate statistics and optimization, which power most of machine learning.


## The intuition

Imagine you have a rubber sheet stretched on a frame. You can push, pull, and twist it, but you never tear it. This is a transformation — it takes every point on the sheet and moves it to a new spot. Now, on that sheet, draw a grid of arrows. After you stretch and twist the sheet, most arrows will point in a completely new direction. But a few special arrows might only get longer or shorter — they do not change direction at all. Those special directions are the eigenvectors, and the amount by which they stretch (or shrink) is the eigenvalue. If an eigenvalue is 2, the arrow doubles in length. If it is 0.5, it halves. If it is 1, it stays the same length. If it is negative, the arrow reverses direction but stays on the same line. Eigenvalues and eigenvectors are the hidden, intrinsic axes of a transformation — the directions that the transformation merely stretches, not rotates. They tell you what the transformation does at its most fundamental level, no matter how you choose to describe the sheet (the coordinate system).


## An analogy

Think of a master chef tasting a soup. She has a palate that recognizes individual flavors — saltiness, sweetness, sourness — and she knows that the soup can be described by how much of each of these fundamental tastes it has. The soup is a complex mixture, but she can decompose it into these basic, non-interfering dimensions. Now, stretch the analogy to a machine learning model. The data — say, measurements of a thousand customers — is the soup. The eigenvectors are the fundamental, independent 'tastes' of the data — the directions of maximum variance, where the data spreads out the most. The eigenvalues are how strong that taste is — how much variance is along that direction. PCA, which we will learn later, is the chef's palate: it finds these eigenvectors and eigenvalues to reduce a complex, high-dimensional soup to a few essential ingredients, discarding the weak flavors (small eigenvalues) that add little. This analogy stops working when you realize that, unlike a chef tasting a single soup, an eigenvector is not a property of the data itself but of the transformation — specifically, a matrix that measures how data points vary relative to each other. The chef's fundamental tastes are fixed; the eigenvectors of this variation matrix are learned from the data and change with it.


## Definition

An eigenvector of a square matrix is a non-zero vector that, when the matrix is applied to it, only changes in length (scales) and does not rotate; the corresponding eigenvalue is the factor by which its length changes.


## How it connects

This is the heart of Linear Algebra, which you have already studied in your library's notes on matrices and linear transformations. An eigenvector is a special case of a linear transformation — one where the output is just a scalar multiple (a version of the vector multiplied by a single number) of the input. It connects directly to your notes on 'Linear Transformations' (the matrix acts on a vector) and 'Matrix Multiplication' (applying the matrix to a vector is a matrix-vector product). It also leads to the 'Singular Value Decomposition', which is a more general version of this idea that works for any matrix, not just square ones. And it is the mathematical engine behind 'Principal Component Analysis', which you will explore later.


## The mathematics

This concept is from the branch of mathematics called linear algebra — the study of vectors (lists of numbers) and the transformations that act on them. We care about this branch because data in machine learning is organized into lists (vectors) and tables (matrices), and models are built from matrix transformations. We need to understand what these transformations do fundamentally. The operation we need to grasp is matrix-vector multiplication. You have already learned how to multiply two numbers: 2 times 3 equals 6. Multiplying a matrix by a vector is just doing several of those scalar multiplications and additions at once. If you have a 2x2 matrix (a table with 2 rows and 2 columns) and a vector (a list of 2 numbers), the result is another vector of 2 numbers. Each number in the result is the dot product of a row of the matrix with the vector: you multiply the first element of the row by the first element of the vector, add the product of the second element of the row and the second element of the vector, and that sum becomes the corresponding entry in the result. Now, the concept of eigenvalues and eigenvectors is a special property of this operation. For a given square matrix, there might be certain non-zero vectors that, when multiplied by the matrix, come out exactly parallel to where they started — they are just multiplied by a single number. That number is the eigenvalue, and that special vector is the eigenvector.

Let A be a square matrix (a table of numbers). Let v be a non-zero vector (a list of numbers). If there is a number λ (lambda) such that:

A * v = λ * v

Then v is an eigenvector of A, and λ is the corresponding eigenvalue. The symbol * means matrix-vector multiplication (which we just did by hand).


## A worked example

```
Take a simple 2x2 matrix called A: A = [[2, 1], [1, 2]]. We will test two candidate vectors to see if they are eigenvectors. A candidate vector is called v1 = [1, 1] and v2 = [1, -1]. We will multiply the matrix by each vector and see what happens.

Step 1: Step 1: Multiply A by v1.  A * v1 = [ (2*1 + 1*1), (1*1 + 2*1) ] = [3, 3].   # This is matrix-vector multiplication. The first entry of the result is the dot product of the first row [2, 1] with the vector [1, 1] (2*1 + 1*1 = 3). The second entry is the dot product of the second row [1, 2] with [1, 1].
Step 2: Step 2: Compare the result to the original vector. The result is [3, 3].   # The original vector was [1, 1]. The result is exactly 3 times the original vector (3 * [1, 1] = [3, 3]). So this vector only stretches by a factor of 3; it does not rotate.
Step 3: Step 3: Conclude that v1 is an eigenvector. Eigenvalue = 3.   # Since A * v1 = 3 * v1, v1 is an eigenvector and 3 is its eigenvalue.
Step 4: Step 4: Now multiply A by v2.  A * v2 = [ (2*1 + 1*-1), (1*1 + 2*-1) ] = [1, -1].   # First entry: dot product of row 1 [2, 1] with [1, -1] is 2*1 + 1*(-1) = 1. Second entry: dot product of row 2 [1, 2] with [1, -1] is 1*1 + 2*(-1) = -1.
Step 5: Step 5: Compare the result to the original vector. The result is [1, -1].   # The original vector was [1, -1]. The result is exactly 1 times the original vector. So this vector does not change at all; it is also an eigenvector, with eigenvalue 1.

Result: The matrix A has two eigenvectors, v1 = [1, 1] with eigenvalue 3, and v2 = [1, -1] with eigenvalue 1. This means the transformation stretches space three times as much along the (1,1) direction as it does along the (1,-1) direction.
```


## From scratch, in Python

```python
import math

# No external libraries needed for this example

def matrix_vector_multiply(matrix, vector):
    """Multiply a square matrix by a vector."""
    result = []
    for row in matrix:
        # row is a list, vector is a list
        sum_products = 0
        for i in range(len(row)):
            sum_products += row[i] * vector[i]
        result.append(sum_products)
    return result

# Our matrix A from the example
A = [[2, 1], [1, 2]]

# Test the first candidate eigenvector
v1 = [1, 1]
result1 = matrix_vector_multiply(A, v1)
print("Result for v1:", result1)
# Output: [3, 3] which is 3 * [1, 1], so eigenvalue is 3

# Test the second candidate eigenvector
v2 = [1, -1]
result2 = matrix_vector_multiply(A, v2)
print("Result for v2:", result2)
# Output: [1, -1] which is 1 * [1, -1], so eigenvalue is 1

```


## The same thing, with a framework

```python
import numpy as np

# Define the matrix
A = np.array([[2, 1], [1, 2]])

# Define candidate eigenvectors
v1 = np.array([1, 1])
v2 = np.array([1, -1])

# Matrix-vector multiplication with numpy
result1 = A.dot(v1)  # or A @ v1
result2 = A.dot(v2)

print(result1)  # [3 3]
print(result2)  # [1 -1]

# Find all eigenvalues and eigenvectors directly
w, v = np.linalg.eig(A)
print("Eigenvalues:", w)  # [3. 1.]
print("Eigenvectors:", v)  # columns are the eigenvectors [[0.707, 0.707], [0.707, -0.707]]

```


## What the code does

This code introduces three Python building blocks. First, a list, which holds values in order and can be changed — for example, `[2, 1]` is a list of two numbers. We use lists to represent vectors (like `[1, 1]`) and rows of a matrix. Second, a for loop, which repeats a block of code for each item in a list — `for row in matrix` runs once for each row in the matrix. Inside that loop, we have a nested loop `for i in range(len(row))` that goes through each position in the row. Third, a function, defined with `def`, which packages code so it can be reused — `matrix_vector_multiply(matrix, vector)` takes a matrix and a vector as inputs and returns the result. The function computes the dot product of each row with the vector: it starts with `sum_products = 0`, then for each position `i`, multiplies the ith element of the row by the ith element of the vector and adds it to the sum. After all multiplications and additions, that sum is appended to the result list. We then call the function twice and print the outputs — the outputs [3, 3] and [1, -1] confirm that our vectors are eigenvectors with eigenvalues 3 and 1.


## Common misconceptions

**Eigenvectors are the same as principal components** — I can put my data matrix directly into an eigensolver and the eigenvectors coming out are the principal components.

Principal components are the eigenvectors of the covariance matrix of the data, not the data matrix itself. The covariance matrix measures how features vary together, and its eigenvectors are the directions of maximum variance. The raw data matrix does not have that geometric meaning. In fact: To get principal components, you first compute the covariance matrix (or use SVD on the centered data), then find its eigenvectors.

**Eigenvalues must be positive** — An eigenvalue is a length, so it cannot be negative.

An eigenvalue can be negative. A negative eigenvalue means the eigenvector is flipped in direction (still on the same line, just pointing the other way). A negative eigenvalue still represents a stretch — by that magnitude — but with an inversion. In fact: Eigenvalue signs matter: the sign tells you about direction reversal, and the magnitude (absolute value) tells you about the stretch factor.

**Every matrix has as many eigenvalues as its size** — A 2x2 matrix always has exactly 2 eigenvalues and 2 eigenvectors.

Some matrices have no real eigenvectors at all (rotations), and some have repeated eigenvalues but not enough eigenvectors. For instance, a pure rotation matrix (like [[0, -1], [1, 0]]) rotates everything by 90 degrees — no non-zero vector stays on the same line, so there are no real eigenvectors. In fact: Over the complex numbers, every matrix has eigenvalues, but they may be complex (like the rotation matrix). Over the real numbers, some matrices have no eigenvectors. Defective matrices have fewer eigenvectors than the size of the matrix.

**Eigenvalues are properties of the data** — The eigenvalues of my covariance matrix are a property of the world, so they will stay the same no matter what data I sample.

The covariance matrix is estimated from the data you have. If you sample different points, the estimated covariance changes, and so do the eigenvalues. The true, underlying distribution has fixed eigenvalues, but your estimate fluctuates. In fact: Eigenvalues of an estimated matrix are themselves noisy estimates. This is why in PCA you often look at the 'scree plot' (sorted eigenvalues) and choose a cutoff that is robust to small fluctuations.

**The largest eigenvalue is always the most useful** — I will just always keep the component with the largest eigenvalue and discard the rest.

In PCA, the largest eigenvalue points to the direction of most variance, but that direction is not always the most useful for your task. For example, a dataset of handwritten digits (0-9) might have its largest variance along a direction that separates '0' from '1', but you need to separate all ten digits, so you need more than one component. In fact: For PCA, you typically keep enough eigenvectors to explain a high fraction (e.g., 95%) of the total variance, not just the single largest. For other tasks, the eigenvector with the largest eigenvalue might not even be the one you want (e.g., in spectral clustering, you often use the eigenvectors associated with the smallest non-zero eigenvalues).

**All matrices can be diagonalized** — Since we learned eigendecomposition, every square matrix has this nice decomposition A = VΛV⁻¹.

The decomposition requires that the matrix has a full set of linearly independent eigenvectors. Many matrices, including some with repeated eigenvalues, are defective and cannot be split into the diagonal form. This is a core reason why SVD exists — it works for all matrices. In fact: Eigendecomposition is a special privilege of certain matrices (diagonalizable ones). SVD is the general tool that works for all matrices, square or not, and is the safer default.


## In short

An eigenvector is a non-zero vector that a transformation (matrix) merely stretches, not rotates. The eigenvalue is the stretch factor. If A·v = λ·v, then v is an eigenvector of A with eigenvalue λ. Eigenvalues can be positive, negative, or zero; negative means the direction flips. The set of all eigenvalues reveals the intrinsic axes of a transformation, independent of the coordinate system. This idea powers PCA (the eigenvectors of a covariance matrix give the directions of maximum variance), underpins PageRank (the dominant eigenvector of a link matrix), enables model compression (only directions with large eigenvalues matter), and stabilizes training via spectral normalization. Most software uses `numpy.linalg.eig` (or `eigh` for symmetric matrices) to compute these, but understanding the meaning is what lets you diagnose and design systems based on that geometry.


## Where this is used

You will not be writing an eigenvalue solver from scratch in your career — but you will be using concepts built on top of it constantly, and you will be calling libraries that compute them. The mechanism, though, is everywhere.

**Principal Component Analysis (PCA) — the number-one user.** PCA is a dimension-reduction technique. Given a dataset of, say, 1000 features per sample, PCA finds the directions of maximum variance. How? It computes the covariance matrix of the data, which measures how each feature varies with every other feature. The eigenvectors of that covariance matrix are the principal components, and the eigenvalues tell you how much variance each component explains. A developer writes `from sklearn.decomposition import PCA; pca = PCA(n_components=5)` and then `pca.fit_transform(X)`. Underneath, Scikit-learn calls `numpy.linalg.eigh` to compute eigenvectors and eigenvalues. You use PCA to visualise high-dimensional data in 2D, to denoise images, to speed up training by reducing feature count, and to find hidden factors. Every time you see a 'variance explained' plot that drops off, you are seeing the sorted eigenvalues.

**Large language models (LLMs) and model compression.** LLMs have billions of parameters (weights). The matrices inside them are huge, and many of the eigenvalues are tiny. This is the basis of low-rank compression and of techniques like LoRA (Low-Rank Adaptation). The idea is that the weight update during fine-tuning has low effective rank — meaning only a few directions (eigenvectors) with large eigenvalues actually matter. LoRA adds a small, low-rank matrix to the weights, effectively approximating the full update in the directions of the dominant eigenvectors. A practitioner imports `from peft import LoraConfig, get_peft_model` and sets `r=8` (the rank). Under the hood, this assumes that the important directions of change are few, which is a statement about eigenvalues. The same logic — `torch.linalg.eigvalsh` — is used in spectral normalization, which clips the eigenvalue magnitudes of weight matrices to stabilize training.

**Small models on devices (TinyML).** When you have to run a neural network on a phone or a microcontroller, you cannot afford a 70-billion-parameter model. You use techniques like pruning (removing near-zero weights) and quantization. Eigenvalue analysis tells you which dimensions of your model's feature space carry signal. For a keyword-spotting model that listens for 'Hey phone', the covariance matrix of the activations of the last hidden layer is analyzed — its top eigenvectors indicate what the model actually uses to make a decision, and you can compress by projecting onto those. In practice, you might use `keras` or `tflite` to convert a model, but the pre-analysis of which components to keep uses eigenvalues from `numpy.linalg`.

**Retrieval-Augmented Generation (RAG).** When an LLM needs to answer a question with specific, current information, it searches a knowledge base. The system embeds both the query and the documents into vectors of, say, 768 dimensions. To find the best match, you compute a similarity score, often the cosine of the angle between vectors. But those 768 dimensions are not equally important. An eigenvalue decomposition of the embedding space can tell you that the first 50 eigenvalues account for 90% of the variance, so you can safely reduce the embedding dimension to 50 (via PCA), making the vector search 15 times faster. The developer does not write the eigen-decomposition; they load a precomputed projection matrix (from `sklearn.decomposition.PCA`) and apply it to the database — the eigenvectors were computed once, offline.

**Agentic systems (planning and tool-calling).** Agents are programs that decide on a sequence of actions. They often use a 'state representation' of the world — a vector of relevant features. When the state space is huge (e.g., all possible sensor readings in a robot), an agent's internal model of the world uses PCA on collected states to reduce dimensionality, so its planning algorithm (often a variant of Monte Carlo Tree Search or a learned policy) operates in a smaller space. The eigenvectors of the state covariance matrix define the 'world embedding'. You might use `sklearn.decomposition.PCA` on the collected state data before feeding it into a policy network.

**Recommendation systems.** Collaborative filtering (the 'people who bought this also bought that' engine) works by decomposing a user-item matrix. The mathematical tool most often used is Singular Value Decomposition (SVD), which is a generalisation of eigenvalues to non-square matrices. The SVD finds the orthogonal directions (eigenvectors of the matrix times its transpose) along which users and items project. A practitioner uses `sklearn.decomposition.TruncatedSVD` to find, say, the top 20 hidden factors, and then computes user-item similarity in that reduced space. The eigenvalues of the 'user covariance' matrix tell you how much preference signal each hidden factor captures.

**Forecasting and anomaly detection.** In time-series analysis, eigenvalues of the covariance matrix of the signal help identify dominant periodic components (via the related technique of multitaper or Karhunen-Loève transform). An anomaly is a point that lies far along a weak eigenvector. In fraud detection, the covariance matrix of normal transactions is built, its eigenvectors give the principal axes, and a transaction far from the data's typical spread (a high Mahalanobis distance) is flagged. The Mahalanobis distance is computed using the inverse of the covariance, which requires its eigenvalues.

**What breaks when it is done badly.** If you ignore eigenvalues and naively keep all dimensions, your model is slow and overfits. If you over-compress and keep too few eigenvectors, you lose signal and accuracy drops. The worst failure is numerical instability — eigenvalues computed with naive algorithms can be wildly wrong; libraries like LAPACK (used by NumPy) employ careful, stable routines, but your own hand-rolled version likely will not. Learn to trust the libraries for the computation, but understand the meaning so you can diagnose why your PCA-variance plot does not drop as expected.


---

Read this on the web: https://basavapurushottam.com/learn-ai/eigenvalues-and-eigenvectors
