# Vector Spaces and Rank

A vector space is a set closed under addition and scaling; rank is the dimension of the space a matrix's columns actually span.

**Leads to:** Determinants and Inverses, Eigenvalues and Eigenvectors, Linear Transformations, Normal Equations, Orthogonality and Projections, PageRank, Singular Value Decomposition, Support Vector Machines, Tensors and Shapes, Vector Native Storage

**Rests on nothing.** A place to start.

*Mathematical Foundations · Linear Algebra*


## Why learn this

This concept reveals how much truly distinct information exists within a collection of data or a model. Understanding it helps you see why some datasets fail to produce useful predictions, how AI systems can hold vast amounts of knowledge without using excessive memory, and why certain changes to a large model require very little computing power. Consider a spreadsheet of customer data where the third column is simply a copy of the first two columns. That third column adds zero new insight because its information was already present. This principle explains how modern AI saves storage space: by identifying and keeping only the essential patterns while discarding redundant noise. It highlights the critical difference between having many numbers on hand and having genuine variety among them.


## The intuition

Imagine trying to describe a location in a city using specific directions. If you say "go 3 blocks east" and "4 blocks north," those two movements are independent; you need both to reach any specific corner of the grid. However, if someone then adds an instruction to "go 6 blocks west," that final step is redundant because it undoes part of the first movement. The rank of a matrix measures exactly this kind of independence.

A matrix is simply a grid of numbers arranged in rows and columns. You can think of each column as a set of instructions or directions. The rank is the count of those columns that are genuinely unique and necessary to describe all possible outcomes. If one column is just a multiple of another—for instance, if Column B is exactly twice Column A—it adds no new direction, so the rank remains low.

For example, in a 3x3 grid (a square of nine numbers), if you only have two unique directions among the columns, the rank is 2. This tells you that despite having three columns, the system can only move within a flat plane, not fill up all of 3D space. You are limited by the number of truly independent inputs, not the total number of inputs.


## An analogy

Think of a paint mixing station with several jars of paint. Each jar contains a specific mixture of red, green, and blue light intensities. The range of colors you can create depends on the jars available. The rank is the minimum number of jars you need to keep on the table to create every color that was originally possible with all the jars. If you have ten jars, but five are just duplicates of the other five mixed at different concentrations, your rank is only five; those extra five jars add no new colors. This analogy works well for understanding redundancy and capability, though it breaks down if you consider that real paint mixing isn't perfectly linear or predictable like mathematical vectors.


## Definition

The rank of a matrix is the number of independent directions its columns provide, which equals the count of truly unique pieces of information within that grid of numbers.


## How it connects

This idea sits at the center of linear algebra. It connects to the determinant: a measure of how much a grid of numbers scales space; if this value is zero, the rank is low and information is lost. It also connects to Singular Value Decomposition: a method that breaks a matrix into its essential parts, where the number of significant parts equals the matrix's rank.


## The mathematics

This idea comes from a branch of mathematics called linear algebra, which is the study of vectors and the rules for combining them. Before we talk about rank, you need to know what a vector is and how we can combine them.

A vector is simply a list of numbers. You can think of it as a point in space, or an arrow pointing in a certain direction. For example, the vector [3, 4] could represent a movement of 3 units to the right and 4 units up.

We can combine vectors in two basic ways:
1. Addition: We add vectors by adding their corresponding numbers. If you have the vector [3, 4] and add the vector [1, 2], you get [4, 6]. In the arrow picture, you place the tail of the second arrow at the head of the first, and the sum is the arrow from the start to the end.
2. Scaling: We can multiply a vector by a number (called a scalar). If you multiply [3, 4] by 2, you get [6, 8]. This simply stretches or shrinks the arrow by that factor. If you multiply by a negative number, the arrow flips direction.

Now, a vector space is a collection of vectors where you can do these two operations (addition and scaling) and the result stays within the collection. For example, the set of all vectors with two numbers (like in a flat plane) is a vector space. The set of all vectors with three numbers is another. The set of vectors that all point along a single line is also a vector space — add any two on that line, and the result is still on the line; scale any of them, and it's still on the line.

A set of vectors is called a 'spanning set' if you can reach any vector in the space by adding and scaling these vectors. For example, the vectors [1, 0] and [0, 1] span the entire plane. You can write any vector [x, y] as x times [1, 0] plus y times [0, 1].

Sometimes, a vector is 'redundant' because you can already get it by combining other vectors. For example, in the set of vectors [1, 0], [0, 1], and [1, 1], the last one is redundant because it's just the sum of the first two. The set { [1, 0], [0, 1] } is said to be 'linearly independent'. The dimension of a vector space is the smallest number of linearly independent vectors that can span it.

Now, consider a matrix, which is a rectangular table of numbers. Its columns are each a vector. The 'column space' is the set of all vectors you can make by adding and scaling these column vectors. The rank of the matrix is the dimension of this column space — the number of genuinely independent columns.

rank(A) equals the dimension of the span of the columns of A


## A worked example

We have the matrix:
  | 1  0 |
  | 0  1 |

Step 1: Column 1 = [1, 0]   # This is the first column vector.
Step 2: Column 2 = [0, 1]   # This is the second column vector.
Step 3: Can [1, 0] be made by scaling [0, 1]? No.   # Scaling [0,1] always gives a vector with first component 0.
Step 4: Can [0, 1] be made by scaling [1, 0]? No.   # Scaling [1,0] always gives a second component 0.
Step 5: These columns are linearly independent.   # Neither can be expressed as a (scalar) multiple of the other.
Step 6: The columns span the whole 2D plane, so the dimension is 2.   # We can reach any vector [x, y] by x*[1,0] + y*[0,1].

Result: The rank of this matrix is 2, because its only columns are linearly independent and span a 2-dimensional space.


## From scratch, in Python

```python
# --- Building a matrix as a list of lists ---
# A matrix is a list of rows. Each row is a list of numbers.
# Let's call the matrix 'A'.

A = [
    [1, 2],
    [3, 4]
]  # This is a 2x2 matrix

# --- Vector space: spanning, independence, rank ---
# We'll write tiny functions to check independence and find the rank.

# We need a helper to find the greatest common divisor (GCD) for simplification.
# But for this basic version, we'll just check linear dependence by scaling.

# First, get the columns of the matrix.
def get_columns(matrix):
    """Return a list of column vectors (each column as a list)."""
    num_rows = len(matrix)
    num_cols = len(matrix[0])
    columns = []
    for j in range(num_cols):
        col = []
        for i in range(num_rows):
            col.append(matrix[i][j])
        columns.append(col)
    return columns

# Check if one vector is a scalar multiple of another (for 2D vectors only)
def is_scalar_multiple(v1, v2):
    """Return True if v1 is a scalar multiple of v2."""
    n = len(v1)
    ratio = None
    for i in range(n):
        if v2[i] == 0:
            # if v2 component is 0, then v1 component must also be 0
            if v1[i] != 0:
                return False
            continue
        if ratio is None:
            ratio = v1[i] / v2[i]
        else:
            if abs(v1[i] / v2[i] - ratio) > 1e-9:
                return False
    if ratio is None:
        return True  # all zeros are multiples
    return True

# Find rank: start with empty set, add columns that are not linear combinations of previous.
def rank_basic(matrix):
    columns = get_columns(matrix)
    independent_cols = []
    for col in columns:
        # Check if col is a linear combination of the current independent set.
        # For simplicity, we only handle 2D: check if col is scalar multiple of any.
        is_dependent = False
        for ind_col in independent_cols:
            if is_scalar_multiple(col, ind_col) or is_scalar_multiple(ind_col, col):
                is_dependent = True
                break
        if not is_dependent:
            independent_cols.append(col)
    return len(independent_cols)

# Test with our matrix A
print("Matrix A:")
for row in A:
    print(row)
print("Rank of A =", rank_basic(A))  # Expect 2

# Now a rank-deficient matrix:
B = [
    [1, 2],
    [2, 4]
]  # Second column is 2x the first
print("\nMatrix B:")
for row in B:
    print(row)
print("Rank of B =", rank_basic(B))  # Expect 1
```


## The same thing, with a framework

```python
import numpy as np

A = np.array([[1, 2], [3, 4]])
B = np.array([[1, 2], [2, 4]])

print("NumPy rank of A:", np.linalg.matrix_rank(A))  # 2
print("NumPy rank of B:", np.linalg.matrix_rank(B))  # 1

# What NumPy does underneath: it computes the singular values and counts
# how many are above a small tolerance.
# Equivalent to our loop over columns but using more advanced techniques.
```


## What the code does

In the from-scratch Python, we build a matrix as a list of lists — a list for the rows, and each row itself a list. We define how to get columns by going through each row and picking the j-th element of every row. The rank_basic function loops over each column and checks if it is a scalar multiple of any previously accepted independent column. If it isn't, we add it to the independent set. The rank is just how many columns we end up with. This is a simplistic version that only works for 2D vectors and only catches linear dependence when one vector is a scalar multiple of another — it misses cases like [1,1] and [2,2] (which it catches) but also misses when a combination like [1,1] and [1,2] are not multiples but span 2D anyway, which would falsely report rank 1. The actual algorithm requires more robust checks like Gaussian elimination. The NumPy version uses the singular value decomposition, which is far more robust and is the standard method in practice.


## Common misconceptions

**Rank is the number of non-zero entries** — Rank is how many numbers in the matrix are not zero.

A matrix with all non-zero entries can have rank 1 (e.g., [[1,2],[2,4]]), while a matrix with many zeros can have full rank (e.g., identity matrix). Rank is about independence, not presence of numbers. In fact: Rank is the dimension of the column space, i.e., the number of linearly independent columns, not the count of non-zero entries.

**Rank is the same as the dimension of the matrix** — A 3x3 matrix always has rank 3.

A 3x3 matrix can have rank 1, 2, or 3. It only has rank 3 if its columns are linearly independent, which is not guaranteed. For example, [[1,2,3],[2,4,6],[3,6,9]] has rank 1. In fact: Rank is at most the smaller of the number of rows and columns, but it can be less. It is the actual dimension of the column space.

**Rank is about the number of rows** — A matrix with more rows has a higher rank.

Rank is limited by both rows and columns. A 10x2 matrix can have rank at most 2, because there are only 2 columns. Adding rows doesn't increase rank beyond the number of columns. In fact: Rank is at most the minimum of the number of rows and columns. It is about how many independent columns exist, not how many rows.

**Rank deficiency means the matrix is 'bad'** — A matrix with low rank is always a failure or an error.

Low-rank matrices are often desirable. In ML, low-rank approximations are used to reduce parameters and improve generalization. The identity matrix has full rank, but a film-rating matrix is naturally low-rank because users' preferences are correlated. In fact: Rank deficiency is a property, not a value judgment. Low-rank structure can be exploited for compression and speed, and it is common in real-world data.

**Rank is the same as the determinant** — If the determinant is zero, the rank is zero.

The determinant is only defined for square matrices, and it being zero tells you that the rank is less than the size of the matrix, but not what it exactly is. A 3x3 matrix of rank 2 has determinant 0, but rank is 2, not 0. In fact: For a square matrix, determinant zero means the matrix is not full rank (rank < size). The rank can be 1, 2, etc., not necessarily 0. The determinant gives a yes/no whether the matrix is full rank.

**Rank is the same as the number of eigenvalues** — A matrix with three eigenvalues has rank 3.

Eigenvalues are only defined for square matrices. Even then, repeated eigenvalues or complex eigenvalues complicate things. The rank is the number of non-zero eigenvalues only if the matrix is diagonalizable, which is not always the case. In fact: Rank is the dimension of the column space. For a diagonalizable matrix, it equals the number of non-zero eigenvalues, but not every matrix is diagonalizable. The SVD is a more reliable tool to compute rank.


## In short

A vector space is a set of vectors closed under addition and scaling. The rank of a matrix is the dimension of the space its columns span — the number of linearly independent columns. Rank tells you the actual carrying capacity of a linear transformation: how much information it preserves, and what it destroys. Low-rank structure is the foundation of model compression (like LoRA), efficient storage, and the mathematical reason why we can approximate huge systems with tiny adapters. When you hear '70-billion-parameter model', remember that those parameters are not 70 billion independent degrees of freedom — the effective rank of the weight matrices is far lower, which is why fine-tuning and compression work.


## Where this is used

Rank and low-rank structure appear throughout modern AI. Here is the mechanism in each area.

**Large Language Models (LLMs):** An LLM stores its knowledge in the weights of its neural network layers. These weight matrices are often of very high dimension but have surprisingly low *approximate* rank. This means the information they encode lies on a lower-dimensional subspace, not randomly scattered in the full space. One direct application is **model compression**: you can decompose a large weight matrix into two smaller matrices whose product closely approximates the original, reducing storage and computation. This is exactly what low-rank factorization does, and it is what allows researchers to shrink models for deployment.

**Small models on devices:** When a model has to run on a phone or a smartwatch, memory and compute are tight. A model with a weight matrix of shape (1000, 1000) has 1,000,000 parameters. If we approximate it with two matrices (1000, 50) and (50, 1000), we use only 100,000 parameters — a tenfold reduction. If the true rank of the original matrix is near 50, the approximation is nearly lossless. This is the mathematical core of pruning and knowledge distillation approaches.

**Retrieval-Augmented Generation (RAG):** RAG systems retrieve relevant documents using embeddings — vectors that represent text in a high-dimensional space. The embedding matrix maps words to these vectors. Understanding vector spaces helps you see that embeddings for related words point in similar directions. Rank concepts appear when you use principal component analysis (PCA) to reduce the dimension of embeddings for efficiency, or when you analyse the inherent dimensionality of the embedding space. The rank of the embedding matrix tells you how many truly independent semantic dimensions exist in your corpus.

**Agentic systems:** Agents that plan and execute tasks often maintain a 'state' that is a vector in a large space. The agent's policy is a linear or nonlinear map from state to action. When training such an agent with reinforcement learning, you use optimization algorithms that update the weight matrices. These updates can be low-rank, meaning the agent only needs to adjust a few independent directions of change. This is the fundamental insight behind **LoRA (Low-Rank Adaptation)**. Instead of updating a full weight matrix of size (d, d), LoRA trains two smaller matrices A (d, r) and B (r, d) where r is very small (often 8 or 16). The update to the original weights is ΔW = A·B, which has rank at most r. This works because, during fine-tuning, the changes to the weights are usually small and lie on a low-dimensional subspace. This is why you can fine-tune a 70-billion-parameter LLM with a small set of trainable parameters.

**Everyday applied ML:** In recommender systems, user-item rating matrices are often sparse and rank-deficient. Matrix factorization techniques (like SVD) are used to find low-rank approximations that predict missing ratings. In computer vision, convolutional filters can be approximated with low-rank factorizations to speed up inference. In fraud detection, features are often high-dimensional but the intrinsic signal is low-rank, and methods like PCA are used to denoise and reduce overfitting.

**What developers write:** When building a system, a developer might call `np.linalg.svd()` to compute the singular value decomposition, or `np.linalg.matrix_rank()` to check for degeneracy. In PyTorch, they might use `torch.linalg.svd()` or set up a LoRA layer that defines two low-rank matrices and multiplies them to update the weights. The most common failure is when a developer forgets to check for rank deficiency in a system, leading to a matrix that cannot be inverted — a classic 'singular matrix' error. Understanding rank helps you diagnose these issues.

**Honest absence:** Rank is a linear algebra concept that does not directly appear in the forward pass of a neural network as a computed quantity; it is more often used in analysis, compression, and training strategies. It is not used directly in the inference of a single forward pass, except in some specialized kernels. But it is fundamental to understanding the geometry of the weight space and why techniques like LoRA work.


---

Read this on the web: https://basavapurushottam.com/learn-ai/vector-spaces-and-rank
