# Singular Value Decomposition

Factoring any matrix into a rotation, a scaling and another rotation.

**Rests on:** Linear Transformations, Matrix Multiplication, Vector Spaces and Rank

**Leads to:** LoRA and QLoRA

*Mathematical Foundations · Linear Algebra*


## Why learn this

You use tables of numbers every day — spreadsheets, customer lists, sensor readings, or even the pixels in a photo. Often, these tables contain more detail than you need, and that extra data makes your files large, your calculations slow, and the underlying patterns hard to see. Singular Value Decomposition lets you compress that data into a smaller, simpler form without losing the meaningful parts. For example, an image file might be megabytes in size because it stores millions of pixel values. SVD can reduce that to a few key numbers while keeping the picture looking nearly identical to the naked eye. This is not just theory; it is how you save storage space, speed up search results, and remove noise from noisy measurements.


## The intuition

Imagine a spreadsheet with 1,000 rows (people) and 500 columns (questions they answered). The numbers inside are ratings from 1 to 5. This table holds complex relationships: perhaps people who rate Sci-Fi high also rate Action high. SVD breaks this huge table down into three simpler steps that describe the same data exactly.

Step 1: Rotate the perspective of the people. Instead of looking at them by their original names, you align them along new axes that capture the strongest agreements in their answers. For instance, one axis might separate 'Sci-Fi fans' from 'Romance fans.'

Step 2: Scale the importance of each axis. You multiply the first axis by a big number (because it explains most of the variance), the second by a medium number, and so on. The smallest numbers correspond to weak or noisy patterns.

Step 3: Rotate the perspective of the questions. Now you align the questions themselves along these new axes.

The magic happens when you stop here. You keep only the top few scaled axes — say, just the first 10 out of the original hundreds. Because you kept the strongest signals and discarded the weakest noise, you can reconstruct a table that is almost indistinguishable from the original one, but it takes up far less space to store because you only need to remember those 10 strong patterns instead of every single rating. If you had 500 dimensions originally, reducing it to 10 is a massive simplification.


## An analogy

Think of SVD as tuning a complex radio signal into its fundamental frequencies. A noisy radio broadcast contains many overlapping tones and static. SVD acts like a perfect filter that identifies the three or four main musical notes being played simultaneously and their relative loudness. It ignores the hiss, the crackle, and the faint background noise (the small scaling factors). If you replay only those main notes, the music is still recognizable and enjoyable, but it is simpler to describe than the entire raw signal. The caveat is that unlike a guitar string which vibrates in specific physical ways, these 'tones' are abstract mathematical directions that might not correspond to any single real-world variable on its own.


## Definition

Singular Value Decomposition is a method to split any rectangular table of numbers into three simpler tables: two that represent pure rotations (rearranging the axes without stretching) and one that represents scaling (stretching along those new axes), such that multiplying them together reconstructs the original table exactly.

Formally, it factors a matrix A into U Σ V^T, where U and V are rotation-like matrices with perpendicular columns, and Σ is a diagonal matrix containing non-negative values called singular values.


## How it connects

This builds directly on your understanding of Matrix Multiplication, which you know composes transformations. SVD reveals that *any* transformation can be decomposed into a rotation, a stretch, and another rotation. It also relates closely to Orthogonality and Projections: the 'pure rotations' in SVD are built from vectors that are perpendicular (orthogonal) to each other, meaning they capture completely independent aspects of the data, unlike overlapping variables that carry redundant information.


## The mathematics

Singular Value Decomposition (SVD) is a way to break down any rectangular table of numbers, called a matrix, into three simpler pieces. This works for every matrix, whether it is square or not, and whether it has more rows than columns or vice versa.

To understand SVD, we first need two ideas: matrix multiplication and orthogonal matrices.

**Matrix Multiplication**
When you multiply two matrices, you are combining their effects. If you have a table of numbers A and another table B, the product AB is calculated by taking each row of A and each column of B, multiplying the matching numbers together, and adding them up. This sum becomes one entry in the new matrix.

**Orthogonal Matrices**
An orthogonal matrix is a special kind of square matrix. Its columns are all perpendicular to each other (at right angles), and each column has a length of exactly 1. Geometrically, multiplying by an orthogonal matrix is like rotating or reflecting space without stretching or shrinking anything. It preserves distances.

**The SVD Formula**
SVD says that any matrix A can be written as the product of three matrices:
A = U * S * V_transpose

Here is what each part means:

1. **V (the right singular vectors)**: This is an n x n orthogonal matrix. Its columns are special directions in the input space. We call them 'right-singular vectors'. Because V is orthogonal, its transpose, written as V_transpose (or V^T), undoes the rotation that V performs.

2. **S (the singular values)**: This is an m x n matrix. It looks like a diagonal matrix, which means it has numbers on the main diagonal from top-left to bottom-right, and zeros everywhere else. The numbers on the diagonal are called 'singular values'. We always list them in order from largest to smallest: sigma_1 >= sigma_2 >= ... >= 0. These values tell us how much to stretch or shrink space along each of the directions defined by V.

3. **U (the left singular vectors)**: This is an m x m orthogonal matrix. Its columns are 'left-singular vectors'. They represent the directions in the output space where the stretched data lands.

**How it works step-by-step**
Imagine you have a vector of numbers x. To transform it using A, SVD breaks the process into three simple steps:

Step 1: Rotate the input.
Multiply x by V_transpose. This rotates the coordinate system so that the most important directions in the data line up with the standard axes (the x-axis, y-axis, etc.).

Step 2: Scale the axes.
Multiply the result by S. Since S is diagonal, this just multiplies each coordinate by a singular value. The largest singular value stretches the first coordinate the most. Smaller singular values stretch later coordinates less. If a singular value is zero, that direction is collapsed to nothing.

Step 3: Rotate the output.
Multiply the result by U. This rotates the scaled data into its final position in the output space.

**Why it is useful**
Because the singular values are ordered from largest to smallest, they tell us which directions carry the most information. If we set the smallest singular values to zero, we get a simplified version of the matrix. This simplified version is the best possible approximation of the original matrix using fewer dimensions. It is like compressing a file while losing the least amount of detail.

**Notation Guide**
- A: The original m x n matrix (m rows, n columns).
- U: An m x m orthogonal matrix. Its columns are u_1, u_2, ..., u_m.
- S: An m x n diagonal matrix with singular values sigma_1, sigma_2, ... on the diagonal.
- V: An n x n orthogonal matrix. Its columns are v_1, v_2, ..., v_n.
- V_transpose: The transpose of V (rows become columns).
- sigma_i: The i-th singular value.


## A worked example

Let us compute the SVD for a small 2x2 matrix A. This matrix might represent ratings from 2 users on 2 movies.

A = [[1, 2],
     [3, 4]]

We will find U, S, and V such that A = U * S * V_transpose.

**Step 1: Compute A_transpose * A**
First, we find the transpose of A. This means we swap rows and columns.
A_transpose = [[1, 3],
               [2, 4]]

Now multiply A_transpose by A:
A_transpose * A = [[1*1 + 3*3, 1*2 + 3*4],
                   [2*1 + 4*3, 2*2 + 4*4]]
                 = [[1 + 9, 2 + 12],
                    [2 + 12, 4 + 16]]
                 = [[10, 14],
                    [14, 20]]

Let us call this matrix B. So B = [[10, 14], [14, 20]].

**Step 2: Find the eigenvalues of B**
The eigenvalues are numbers lambda such that when we subtract lambda from the diagonal entries and find the determinant, it equals zero.
The characteristic equation is:
det(B - lambda * I) = 0
where I is the identity matrix [[1, 0], [0, 1]].

B - lambda * I = [[10 - lambda, 14],
                  [14, 20 - lambda]]

The determinant is:
(10 - lambda)(20 - lambda) - (14 * 14) = 0
200 - 10*lambda - 20*lambda + lambda^2 - 196 = 0
lambda^2 - 30*lambda + 4 = 0

We solve this quadratic equation using the formula:
lambda = (30 +/- sqrt(30^2 - 4*1*4)) / 2
lambda = (30 +/- sqrt(900 - 16)) / 2
lambda = (30 +/- sqrt(884)) / 2
sqrt(884) is approximately 29.73.

lambda_1 = (30 + 29.73) / 2 = 59.73 / 2 = 29.865
lambda_2 = (30 - 29.73) / 2 = 0.27 / 2 = 0.135

(Note: The original text used approximations 29.07 and 0.93 which were incorrect for this specific matrix A=[[1,2],[3,4]]. The correct eigenvalues of [[10,14],[14,20]] are roughly 29.86 and 0.14. We will use the more precise values here to ensure mathematical correctness.)

**Step 3: Compute the singular values**
The singular values (sigma) are the square roots of the eigenvalues.
sigma_1 = sqrt(29.865) ≈ 5.465
sigma_2 = sqrt(0.135) ≈ 0.367

So the matrix S is:
S = [[5.465, 0],
     [0, 0.367]]

**Step 4: Find the right-singular vectors (columns of V)**
The columns of V are the eigenvectors of B corresponding to lambda_1 and lambda_2.

For lambda_1 = 29.865:
We solve (B - 29.865 * I) * v_1 = 0.
[[10 - 29.865, 14], [14, 20 - 29.865]] * [x, y]^T = [0, 0]^T
[[-19.865, 14], [14, -9.865]] * [x, y]^T = [0, 0]^T

From the first row: -19.865*x + 14*y = 0 => 14*y = 19.865*x => y = (19.865/14)*x ≈ 1.419*x.
Let x = 1, then y ≈ 1.419.
The vector is [1, 1.419].
To make it a unit vector (length 1), we divide by its length.
Length = sqrt(1^2 + 1.419^2) = sqrt(1 + 2.014) = sqrt(3.014) ≈ 1.736.
v_1 = [1/1.736, 1.419/1.736] ≈ [0.576, 0.817].

For lambda_2 = 0.135:
We solve (B - 0.135 * I) * v_2 = 0.
[[10 - 0.135, 14], [14, 20 - 0.135]] * [x, y]^T = [0, 0]^T
[[9.865, 14], [14, 19.865]] * [x, y]^T = [0, 0]^T

From the first row: 9.865*x + 14*y = 0 => 14*y = -9.865*x => y = (-9.865/14)*x ≈ -0.705*x.
Let x = 1, then y ≈ -0.705.
The vector is [1, -0.705].
Length = sqrt(1^2 + (-0.705)^2) = sqrt(1 + 0.497) = sqrt(1.497) ≈ 1.223.
v_2 = [1/1.223, -0.705/1.223] ≈ [0.818, -0.576].

So V is:
V = [[0.576, 0.818],
     [0.817, -0.576]]

**Step 5: Find the left-singular vectors (columns of U)**
The columns of U are found by multiplying A by the columns of V and dividing by the corresponding singular value.
u_i = (A * v_i) / sigma_i

For u_1:
A * v_1 = [[1, 2], [3, 4]] * [0.576, 0.817]^T
= [1*0.576 + 2*0.817, 3*0.576 + 4*0.817]^T
= [0.576 + 1.634, 1.728 + 3.268]^T
= [2.210, 4.996]^T

u_1 = [2.210 / 5.465, 4.996 / 5.465]^T ≈ [0.404, 0.914]^T.

For u_2:
A * v_2 = [[1, 2], [3, 4]] * [0.818, -0.576]^T
= [1*0.818 + 2*(-0.576), 3*0.818 + 4*(-0.576)]^T
= [0.818 - 1.152, 2.454 - 2.304]^T
= [-0.334, 0.150]^T

u_2 = [-0.334 / 0.367, 0.150 / 0.367]^T ≈ [-0.910, 0.409]^T.

So U is:
U = [[0.404, -0.910],
     [0.914, 0.409]]

**Step 6: Verify the result**
We check if U * S * V_transpose equals A.

First, compute S * V_transpose:
S * V_transpose = [[5.465, 0], [0, 0.367]] * [[0.576, 0.817], [0.818, -0.576]]
= [[5.465*0.576, 5.465*0.817], [0.367*0.818, 0.367*(-0.576)]]
≈ [[3.148, 4.466], [0.300, -0.211]]

Now multiply U by this result:
U * (S * V_transpose) = [[0.404, -0.910], [0.914, 0.409]] * [[3.148, 4.466], [0.300, -0.211]]

Entry (1,1): 0.404*3.148 + (-0.910)*0.300 = 1.272 - 0.273 = 0.999 ≈ 1
Entry (1,2): 0.404*4.466 + (-0.910)*(-0.211) = 1.804 + 0.192 = 1.996 ≈ 2
Entry (2,1): 0.914*3.148 + 0.409*0.300 = 2.877 + 0.123 = 3.000 ≈ 3
Entry (2,2): 0.914*4.466 + 0.409*(-0.211) = 4.082 - 0.086 = 3.996 ≈ 4

The result is [[1, 2], [3, 4]], which matches A.

**Conclusion**
The SVD of A is:
U = [[0.404, -0.910], [0.914, 0.409]]
S = [[5.465, 0], [0, 0.367]]
V = [[0.576, 0.818], [0.817, -0.576]]

The first singular value (5.465) is much larger than the second (0.367). This means the first direction carries most of the information. If we kept only the first term (u_1 * sigma_1 * v_1_transpose), we would get a simplified matrix that approximates A well.


## From scratch, in Python

```python
import statistics

def dot_product(list1, list2):
    """Multiply matching entries and add them up."""
    total = 0
    for i in range(len(list1)):
        total = total + list1[i] * list2[i]
    return total

# Represent a small matrix as a list of lists: each row is a list.
# Our data: 2 users x 2 movies
A = [[1, 2], [3, 4]]

# Step 1: Compute A^T * A (a 2x2 matrix)
  # Transpose A: swap rows and columns
AT = [[A[row][col] for row in range(len(A))] for col in range(len(A[0]))]
# This is a list comprehension - it builds AT by reading down each column of A.

# Compute the product AT * A (each entry is a dot product)
ATA = [[dot_product(AT[i], [A[j][k] for j in range(len(A))]) for k in range(len(A[0]))] for i in range(len(AT))]
# This is complex nesting; for a 2x2 matrix it's simpler to see it directly:
# ATA[0][0] = 1*1 + 3*3 = 10, etc.

# Step 2: For a 2x2 matrix, eigenvalues can be found with the quadratic formula.
# Eigenvalues of
# | a - l    b   |
# |   c    d - l |
# are solutions to l^2 - (a+b)l + (ad - bc) = 0

a = ATA[0][0]
b = ATA[0][1]
c = ATA[1][0]
d = ATA[1][1]

import math
# Solve for l
discriminant = math.sqrt((a + d)**2 - 4 * (a*d - b*c))
lambda1 = ((a + d) + discriminant) / 2
lambda2 = ((a + d) - discriminant) / 2

singular_values = [math.sqrt(max(0, lambda1)), math.sqrt(max(0, lambda2))]
print("Singular values:", singular_values)

# For brevity, we skip finding the eigenvectors by hand here, but we could.
# The full code would compute U, S, V. This is just to show the arithmetic.
```


## The same thing, with a framework

```python
import numpy as np

A = np.array([[1, 2], [3, 4]])
U, S, Vt = np.linalg.svd(A)
print("U:", U)
print("Singular values:", S)
print("V^T:", Vt)
# Reconstruct A from U, S, Vt:
# Note: S is a 1D array of singular values; need to make it a diagonal matrix
Sigma = np.zeros((U.shape[1], Vt.shape[0]))
for i in range(len(S)):
    Sigma[i, i] = S[i]
A_reconstructed = U.dot(Sigma).dot(Vt)
print("Reconstructed A:", A_reconstructed)
```


## What the code does

This code shows the beginning of computing an SVD from scratch. First, we define a function `dot_product` that takes two lists and returns the sum of the products of their matching elements – the dot product. We represent the matrix A as a list of lists, where each inner list is a row. Then we compute A^T (transpose) using a list comprehension: `A[row][col]` reads the element at row and column, and we build a new matrix where rows become columns. Next, we compute the product A^T * A by taking dot products of rows of A^T with columns of A – but the code uses a nested comprehension, which is dense. For a clear view, you could manually write the 2x2 multiplication. After that, we find the eigenvalues of this 2x2 matrix using the quadratic formula, because the eigenvalues of A^T * A are the squares of the singular values. We take square roots to get the singular values. This is the core arithmetic; a full implementation would also compute eigenvectors to form V and then U. This simple code shows you the steps without relying on a library.


## Common misconceptions

**SVD is only for square matrices** — I thought SVD only worked on square matrices, since eigen-decomposition does.

SVD works on any rectangular matrix, no matter the number of rows and columns. This is its power over eigen-decomposition. In fact: SVD applies to all matrices. You can factor a 1000x500 matrix just as easily as a 5x5 one.

**Singular values are always positive** — I think singular values can be negative, like eigenvalues.

Singular values are the square roots of eigenvalues of A^T * A, so they are always non-negative by definition. In fact: Singular values are always zero or positive. They represent the 'amount of stretching' in each direction, and you cannot stretch by a negative amount.

**SVD is an approximation** — SVD is an approximate technique that loses information.

The full SVD is exact – multiplying U * S * V^T gives you back the original matrix exactly. It's only when you truncate (set small singular values to zero) that it becomes an approximation. In fact: SVD is exact. The approximation happens only when you choose to keep a subset of the singular values for compression.

**U and V are the same matrix** — I thought U and V were the same for any matrix.

U and V are different orthogonal matrices that rotate in the row and column spaces respectively. They are only the same for symmetric matrices. In fact: U and V are separate rotations. U rotates in the output space, V rotates in the input space, and S scales between them.

**SVD is only useful for data compression** — SVD is only for making files smaller.

SVD is used for many things: denoising, finding latent factors, dimensionality reduction, solving linear systems, and revealing the intrinsic rank of a matrix. In fact: SVD is a cornerstone of numerical linear algebra, with applications far beyond compression.


## In short

Singular Value Decomposition factors any matrix into U (rotation), S (scaling with non-negative singular values), and V^T (another rotation). It is exact, works for rectangular matrices, and truncating S gives the best low-rank approximation in the least-squares sense. SVD underpins PCA, recommender systems, model compression (LoRA), and many numerical algorithms.


## Where this is used

Singular Value Decomposition is everywhere in AI. In **large language models (LLMs)**, SVD is used for dimensionality reduction and for compressing the weight matrices in neural networks. For example, a technique called Truncated SVD is used to approximate a large weight matrix with a low-rank one, reducing the number of parameters while preserving most of the model's performance. This is the core idea behind **LoRA (Low-Rank Adaptation)**, where instead of updating the full weight matrix during fine-tuning, you learn a low-rank update that is expressed as the product of two smaller matrices – essentially an SVD-like decomposition. In **small models running on devices** like phones, SVD is used to compress model weights so that they fit in limited memory and run faster. A 1000x1000 matrix can be approximated by a product of a 1000x10 and a 10x1000 matrix, dropping from 1 million parameters to 20,000 – a 50x reduction. In **retrieval-augmented generation (RAG)**, embeddings of documents are high-dimensional vectors; SVD (or PCA, which is SVD applied to the covariance matrix) reduces their dimension so that similarity search is faster and more robust to noise. In **agentic systems** that plan and call tools, SVD can be used to compress the state space or the belief space in reinforcement learning, making the planning problem tractable. In **everyday applied machine learning**, SVD is the key to **recommender systems**: the Netflix Prize was won by matrix factorisation, which is exactly SVD where you only keep the top few singular values to predict missing entries in a sparse ratings matrix. It is also used in **image compression** (like JPEG's cousin, SVD compression), **forecasting**, **fraud detection** (finding unusual patterns in transaction matrices), and **speech processing** (separating signals). A developer typically calls `numpy.linalg.svd` or `sklearn.decomposition.TruncatedSVD` rather than implementing it from scratch, but understanding it lets them choose how many components to keep and what the trade-offs are.


---

Read this on the web: https://basavapurushottam.com/learn-ai/singular-value-decomposition
