# Orthogonality and Projections

Two vectors are orthogonal when their dot product vanishes; a projection drops a vector perpendicularly onto a subspace.

**Rests on:** Matrix Multiplication, Vector Spaces and Rank

**Leads to:** Gram-Schmidt, Normal Equations

*Mathematical Foundations · Linear Algebra*


## Why learn this

Imagine you are trying to find the perfect recipe for a cake based on past baking experiments. You have data: the amount of sugar used, the baking time, and the final taste score. The relationship isn't perfect—sometimes the cake is dry even with enough sugar—but there is a general trend. This topic gives you the mathematical tool to find that single best-trend line through scattered data points. It allows you to predict outcomes accurately by filtering out the noise, which is exactly what happens when your phone suggests the next word, when a search engine finds documents similar to your query, or when AI systems learn from vast amounts of information. Without this tool, we could only fit data perfectly on random points, which is useless for prediction.


## The intuition

Think of a flagpole standing on flat ground. The pole itself represents a vector—a direction and length in space. The ground represents a subspace, which you can picture as a flat room where if you draw any straight line across the floor, that entire line stays within the floor; it never breaks through the ceiling or walls. When sunlight shines down vertically, the pole casts a shadow on the floor. That shadow is the projection of the pole onto the ground. It is the part of the pole’s direction that lies entirely within the flat surface of the floor. The rest of the pole—the part sticking straight up into the air—is perpendicular to the floor and has no component lying within it. In mathematics, we often have a complex vector (like a high-dimensional description of a word or an image) and a simpler subspace (like a specific set of meaningful directions). Projecting the vector onto the subspace means finding the shadow it would cast if light shone perpendicularly from above. This is useful because it lets us approximate a complicated object by a simpler one that lives in a smaller space, keeping only the part that matters relative to our chosen directions and discarding the rest. For example, if you have two vectors (arrows), you can measure how aligned they are using a calculation called the dot product. You multiply their corresponding components together and add up the results. If the dot product is zero, the arrows are at right angles to each other—they share no common direction. This state of being perpendicular is what mathematicians call orthogonal. In machine learning, we constantly have high-dimensional data points, and we want to express them in terms of a few important directions. Projection is the tool that collapses the complex data onto these key directions, allowing us to simplify the problem while minimizing error.


## An analogy

Imagine you are standing in a large empty hall with a long, straight railing running across the floor. You hold a laser pointer and shine it at a point on the ceiling. The beam of light travels in a straight line from your hand to that specific spot on the ceiling. Now, imagine there is a second identical railing directly above the first one, attached to the ceiling. If you want to know which point on the upper railing corresponds most closely to your target on the ceiling, you look for the point where a line drawn perpendicular (at a ninety-degree angle) from the ceiling spot would hit the upper railing. That point is the projection of the ceiling spot onto the upper railing. The distance between your original target and this projected point represents the error or 'residual'—the part of your signal that cannot be explained by the direction of the railings. This geometric relationship holds true even when you cannot visualize it: in higher dimensions, projecting a vector onto a subspace is always finding the closest point in that subspace to the original vector, measured by the shortest possible perpendicular path.


## Definition

Two vectors are orthogonal when they are at right angles to each other, meaning their dot product—the sum of the products of their corresponding components—is zero. A projection of a vector onto a subspace is the closest point within that subspace to the original vector, found by moving along the perpendicular direction until you hit the subspace.


## How it connects

This concept sits alongside two neighboring topics: least squares, which is the process of finding the best-fit line or plane by minimizing the sum of squared perpendicular errors between data points and the fit; and singular value decomposition (SVD), which is a method for breaking down complex matrices into simpler orthogonal components to reveal the underlying structure of data.


## The mathematics

This concept lives in the branch of mathematics called linear algebra, which is the study of vectors and the spaces they span. We start with a vector, which is just a list of numbers. For example, the point (3,4) is a vector.

The dot product is a way to multiply two vectors to get a single number. You multiply each matching pair of entries and add the results. The dot product has a geometric meaning: it measures how much two vectors point in the same direction. If the dot product is zero, the vectors are perpendicular—they have no common component. This state is called orthogonality.

Projection is the operation of taking a vector and 'dropping' it perpendicularly onto another vector or a subspace. The result is the shadow of the first vector cast onto the second. The light source is directly above, so the shadow falls straight down. The formula uses the dot product to find this shadow.

To find the projection of vector a onto vector b, follow these steps:
1. Calculate the dot product of a and b. This tells us how much a aligns with b.
2. Calculate the dot product of b with itself. This gives the squared length of b.
3. Divide the result from step 1 by the result from step 2. This quotient is a scalar number. It tells us how many times longer or shorter the shadow is compared to b.
4. Multiply vector b by this scalar number. The result is the projection vector.

This projection vector lies on the line defined by b and is the point on that line closest to a. The difference between the original vector a and its projection is called the error vector. This error vector is perpendicular to b. This property—that the error is orthogonal to the target space—is the foundation of least squares regression and dimensional reduction.

The formula for the projection of a onto b is:
projection = (a dot b divided by b dot b) multiplied by b.

The condition for orthogonality between two vectors x and y is that their dot product equals zero: x dot y = 0.

The perpendicular component of a relative to b is found by subtracting the projection from the original vector:
perpendicular component = a minus projection.


## A worked example

```text
Let a = (3, 4) and b = (2, 0). We will compute the dot products, check orthogonality, and find the projection of a onto b.

Step 1: Compute the dot product of a and b.
Multiply matching entries and sum them:
3 times 2 equals 6.
4 times 0 equals 0.
Sum: 6 + 0 = 6.
So, a dot b = 6.

Step 2: Compute the dot product of b with itself (b squared).
Multiply matching entries and sum them:
2 times 2 equals 4.
0 times 0 equals 0.
Sum: 4 + 0 = 4.
So, b dot b = 4.

Step 3: Compute the scalar multiplier.
Divide the result from Step 1 by the result from Step 2:
6 divided by 4 equals 1.5.
This number tells us that the shadow is 1.5 times as long as vector b.

Step 4: Compute the projection vector.
Multiply vector b by the scalar from Step 3:
1.5 times (2, 0).
1.5 times 2 equals 3.
1.5 times 0 equals 0.
The projection is (3, 0).

Step 5: Compute the perpendicular component (the error).
Subtract the projection from the original vector a:
a minus projection = (3, 4) minus (3, 0).
3 minus 3 equals 0.
4 minus 0 equals 4.
The perpendicular component is (0, 4).

Step 6: Verify orthogonality.
Check if the perpendicular component is orthogonal to b by computing their dot product:
(0, 4) dot (2, 0).
0 times 2 equals 0.
4 times 0 equals 0.
Sum: 0 + 0 = 0.
Since the dot product is zero, the vectors are perpendicular.

Result: The projection of a onto b is (3, 0). The perpendicular component is (0, 4), which is orthogonal to b.
```


## From scratch, in Python

```python
def dot(u, v):
    s = 0
    for i in range(len(u)):
        s += u[i] * v[i]
    return s

def project(u, v):
    scalar = dot(u, v) / dot(v, v)
    return [scalar * x for x in v]

a = [3, 4]
b = [2, 0]
p = project(a, b)
print("projection:", p)  # [3.0, 0.0]
print("orthogonal check:", dot([a[i]-p[i] for i in range(len(a))], b))  # 0.0
```


## What the code does

The functions dot and project are defined. dot loops over indices, multiplying and summing. project computes the scalar and multiplies each component of v. The example uses a = [3,4] and b = [2,0] to show projection and the orthogonality check.


## Common misconceptions

**Orthogonality means the vectors are unrelated** — If two vectors are orthogonal, they have nothing to do with each other, like two completely different topics.

Orthogonality is a geometric relation, not a measure of conceptual relatedness. Two vectors can be orthogonal in 3D yet both be about the same concept. For example, the x-axis and y-axis are orthogonal but both describe position in the plane. In embeddings, two orthogonal vectors share no linear relationship, but they could still co-occur in texts. In fact: Orthogonal means perpendicular—they have no component in common. They are independent directions, but can still relate in context.

**Projection always reduces length** — Projection always makes a vector shorter, because it's a shadow.

If the vector is already parallel to the direction of projection, the projection is the same length as the original. The shadow can be as long as the pole if the sun is directly above and the pole is vertical. In fact: Projection can be equal in length to the original if the vector lies entirely in the subspace. It never exceeds the original length (by Cauchy-Schwarz), but can be equal.

**The dot product must be positive for vectors to be related** — A dot product of zero means the vectors are completely unrelated, and a negative dot product means they are negatively related but still aligned.

The dot product measures the cosine of the angle. Zero means 90°, which is perpendicular—not 'unrelated' in a semantic sense. Negative means the vectors point in opposite directions, which is a strong relationship (opposite orientation). In fact: The sign of the dot product indicates whether the vectors point in the same (positive), opposite (negative), or perpendicular (zero) direction.

**Projection is the same as regression fit** — Projection is just drawing a line through data points, any line that minimizes distance.

Least squares is a specific type of projection: it minimizes the sum of squared perpendicular distances. Not any line does that—only the one that is perpendicular to the residual vector (the errors) in the 'data' space. Projection is the geometric operation; regression is the application. In fact: Least squares is exactly a projection onto the column space of the design matrix, with the residual orthogonal to that space.

**Orthogonal bases are rare and exotic** — Most bases in real data are non-orthogonal, so orthogonality is only a theoretical nicety.

Orthogonal bases are everywhere: the coordinate axes are orthogonal, and methods like PCA and SVD deliberately produce them. In machine learning, we often transform data so that features become orthogonal (whitening). In fact: Orthogonal bases are standard practice for stability and independence of coefficients.


## In short

Orthogonality means perpendicular: dot product zero. Projection is the perpendicular shadow. Projection formula: (a·b / b·b) * b. Least squares = projection. Used everywhere in AI for similarity, compression, and stability.


## Where this is used

Orthogonality and projection are not abstract—they are the machinery behind many AI systems. In large language models, when you embed words as vectors, the similarity between two words is often computed as the cosine of the angle between them, which is the dot product divided by the lengths. If the dot product is zero, the words are orthogonal—they have nothing to do with each other. In retrieval-augmented generation (RAG), a query is embedded into a vector, and you search for documents whose embeddings have high dot products with it. That's projecting the query onto the space of documents and picking the ones with the largest shadows. In small models running on a phone, you often compress the weight matrices using techniques like SVD or PCA, which rely on orthogonal bases. You project the original weight matrix onto a lower-dimensional subspace, keeping the directions of greatest variance. This is a direct application of projection. In agentic systems that plan and call tools, you might need to decide whether a tool is relevant to a current goal. You embed the goal and the tool descriptions, and compute a dot product—if it's high, the tool is relevant; if orthogonal, it's irrelevant. In everyday applied machine learning, least squares is ubiquitous: fitting a line, a logistic regression, or a neural network's final layer all involve projecting the target onto the feature space. The normal equations, which you will meet in linear regression, are exactly the perpendicularity condition we derived. A developer using NumPy might write `np.linalg.lstsq(X, y, rcond=None)` which solves the least-squares problem by finding the projection of y onto the column space of X. Under the hood, it uses QR decomposition or SVD, but the concept is the same: the solution is the orthogonal projection. If you do this badly, you might use an ill-conditioned basis that isn't orthogonal, causing numerical instabilities. That's why Gram-Schmidt (which you'll learn later) is important: it creates orthogonal bases that make computations stable. In models, orthogonal initialisation is sometimes used to keep gradients stable, because orthogonal matrices preserve lengths. In recommendation systems, collaborative filtering uses matrix factorisation, where you project users and items into a lower-dimensional space. The dot product of the user vector and item vector predicts the rating—again a projection. In computer vision, PCA is used for dimensionality reduction before feeding features into a classifier. The principal components are orthogonal directions of maximum variance, and projecting data onto them reduces noise while preserving signal. In natural language processing, word2vec and similar embeddings capture semantic relationships; the classic example is that 'king' - 'man' + 'woman' ≈ 'queen', which works because the vectors are arranged such that the differences align in orthogonal directions. In speech recognition, acoustic feature vectors are often decorrelated using techniques like PCA to make them orthogonal, improving the performance of the downstream model. So, every time you see a dot product or a projection in AI, you're seeing this concept in action.


---

Read this on the web: https://basavapurushottam.com/learn-ai/orthogonality-and-projections
