# Linear Transformations

Maps that preserve addition and scalar multiplication.

**Rests on:** Vector Spaces and Rank

**Leads to:** Activation Functions, Determinants and Inverses, Eigenvalues and Eigenvectors, Gram-Schmidt, Linear Programming and Simplex, Matrix Calculus and Jacobians, Matrix Multiplication, Scattering Transforms, Singular Value Decomposition

*Mathematical Foundations · Linear Algebra*


## Why learn this

Understanding linear transformations lets you see matrix multiplication not as a tedious row-by-column ritual, but as the composition of simple shape shifts. This perspective is essential because every layer in a neural network begins with a linear transformation followed by a nonlinearity—a rule that bends space to make patterns visible. By mastering this idea now, you gain the foundation for grasping how machines process data. Specifically, it clarifies what determinants and inverses measure: how much a transformation stretches or shrinks area and whether that stretching can be perfectly reversed. It also reveals why eigenvalues and eigenvectors matter: they identify the specific directions in space that a transformation simply stretches without rotating, which is key to reducing complex data into its most important components. Finally, it explains the role of singular value decomposition, which breaks any complex shape shift into a rotation, a scaling, and another rotation—the mathematical basis for compressing information and filtering noise.


## The intuition

A linear transformation is a rule that moves every point in space to a new location, but it must follow three strict rules: straight lines must remain straight, parallel lines must remain parallel, and the starting point (the origin) must stay exactly where it is. Think of a grid drawn on a perfectly flexible rubber sheet that is pinned at its center. You can stretch, squash, rotate, or flip the sheet, but you cannot tear it, fold it, or bend it into curves. Because the origin is pinned, any such movement is entirely determined by what happens to just two simple arrows: the arrow pointing one unit to the right and the arrow pointing one unit up. Since every other point in space is just a combination of these two arrows, knowing where they land tells you exactly where every other point goes. This is why a 2x2 table of numbers (a matrix) can completely describe the transformation: it simply records the new locations of those two fundamental arrows.


## An analogy

Imagine a graph-paper grid drawn on a rubber sheet, pinned firmly at the bottom-left corner (the origin). A linear transformation is any way you can stretch, squash, rotate, flip, or shear that sheet without tearing it, folding it, or moving the pin. After the movement, the grid lines are still straight, parallel lines remain parallel, and the origin is still fixed. The entire rule for moving every point is captured by where two particular points land: the point (1,0)—one square to the right—and the point (0,1)—one square up. Suppose (1,0) lands at (2,1) and (0,1) lands at (-1,3). Then the point (2,3), which is 2 copies of the right-arrow plus 3 copies of the up-arrow, must land at 2 copies of (2,1) plus 3 copies of (-1,3). The calculation follows two steps: first, multiply the components: 2 times (2,1) gives (4,2), and 3 times (-1,3) gives (-3,9). Second, add the results: (4,2) + (-3,9) equals (1,11). This arithmetic is exactly how matrices work. The analogy holds perfectly for this type of rigid, origin-fixed stretching. However, note that in neural networks, we often allow the sheet to slide slightly before bending it; that initial slide does not change the shape but shifts the position, a step handled separately by a bias term.


## Definition

A linear transformation is a rule that maps every input vector to an output vector such that the transform of a sum equals the sum of the transforms, and the transform of a scaled input equals the scale times the transform of the input. Formally, it is a function T where T(u + v) = T(u) + T(v) and T(cu) = cT(u) for all vectors u, v and scalars c.


## How it connects

You can think of a vector as an arrow from the origin to a specific point, or simply as a list of coordinates like (2,3). Linear transformations are the core operation in linear algebra, the branch of mathematics that studies these arrows and how they move. The next logical step is matrix multiplication, which is merely a compact notation for applying a linear transformation to every point simultaneously. This leads directly to determinants, which calculate how much area or volume the transformation stretches or shrinks, and eigenvectors, which identify the specific directions that are only stretched, never rotated.


## The mathematics

A linear transformation is a rule that takes an input arrow (a vector) and produces an output arrow, while keeping the straight lines straight and the origin fixed. To understand this rule, we must first look at what it means for a rule to be "linear." A rule, which we will call T, is linear if it obeys two simple habits:

1. It respects addition. If you take two arrows, add them together to make a new arrow, and then apply the rule T to that new arrow, the result is exactly the same as applying the rule T to each of the original arrows separately and then adding those two results.
   In symbols: T(x + y) = T(x) + T(y).

2. It respects stretching (scaling). If you stretch an arrow by a number factor before applying the rule, the result is the same as applying the rule first and then stretching the output by that same factor.
   In symbols: T(c * x) = c * T(x), where c is any number.

Because of these two habits, a linear transformation is completely determined by what it does to just two special arrows: the arrow pointing one unit to the right, which we call e1 = (1, 0), and the arrow pointing one unit up, which we call e2 = (0, 1). These are called the standard basis vectors.

Any other arrow x can be built by combining these two. For example, the arrow x = (x1, x2) is made of x1 copies of e1 and x2 copies of e2. We write this as:
x = x1 * e1 + x2 * e2.

Since the transformation T respects both addition and stretching, we can find where x goes by doing the same combination with the outputs of e1 and e2. Let us call the output of e1 the vector a1, and the output of e2 the vector a2. Then:
T(x) = T(x1 * e1 + x2 * e2)
     = x1 * T(e1) + x2 * T(e2)   (because T respects addition and stretching)
     = x1 * a1 + x2 * a2.

This formula tells us everything. If we know where the two basis arrows go, we know where every arrow goes.

A matrix is simply a compact way to write down these two output vectors, a1 and a2, as columns in a table. If A is the matrix whose first column is a1 and second column is a2, then the operation of applying the transformation T to x is exactly the same as multiplying the matrix A by the vector x.

The multiplication works like this: take the first number of x (which is x1) and multiply it by the first column of A. Take the second number of x (which is x2) and multiply it by the second column of A. Add those two resulting vectors together. The result is the new vector.


## A worked example

Let us follow a specific example step by step.

Suppose our transformation T has the following effect on the basis arrows:
- The arrow (1, 0) maps to the arrow a1 = (2, 1).
- The arrow (0, 1) maps to the arrow a2 = (-1, 3).

We want to find where the input vector x = (2, 3) goes.

Step 1: Break the input vector into its basis components.
The vector x = (2, 3) is made of 2 copies of (1, 0) and 3 copies of (0, 1).
So, x = 2 * (1, 0) + 3 * (0, 1).

Step 2: Identify the output vectors for the basis arrows.
From our definition above:
- The output for (1, 0) is a1 = (2, 1).
- The output for (0, 1) is a2 = (-1, 3).

Step 3: Apply the linearity rule.
Because T is linear, the output for x is the same combination of the outputs:
T(x) = 2 * a1 + 3 * a2.

Step 4: Perform the arithmetic.
First, multiply the scalar 2 by the vector a1:
2 * (2, 1) = (2*2, 2*1) = (4, 2).

Next, multiply the scalar 3 by the vector a2:
3 * (-1, 3) = (3*-1, 3*3) = (-3, 9).

Finally, add these two resulting vectors together:
(4, 2) + (-3, 9) = (4 + (-3), 2 + 9) = (1, 11).

Result: The input vector (2, 3) maps to the output vector (1, 11).


## From scratch, in Python

```python
# We'll represent a linear transformation by its matrix A, and a vector by a list of two numbers.
# The transformation is: output = A multiplied by input.
# For a 2x2 matrix, this is: output[0] = A[0][0]*input[0] + A[0][1]*input[1]
#                              output[1] = A[1][0]*input[0] + A[1][1]*input[1]

def apply_transformation(A, v):
    """Return the result of applying matrix A to vector v."""
    # v is a list like [x, y]. A is a list of two lists, each of length 2.
    # Compute each component of the output using the matrix-vector product formula.
    result = [0, 0]
    result[0] = A[0][0] * v[0] + A[0][1] * v[1]
    result[1] = A[1][0] * v[0] + A[1][1] * v[1]
    return result

# Example: our transformation from earlier, where (1,0) -> (2,1) and (0,1) -> (-1,3)
A = [[2, -1], [1, 3]]
v = [2, 3]

output = apply_transformation(A, v)
print(output)  # Should print [1, 11]
```


## The same thing, with a framework

```python
import numpy as np

A = np.array([[2, -1], [1, 3]])
v = np.array([2, 3])

output = A @ v  # matrix-vector multiplication
print(output)  # [1 11]
```


## What the code does

The code defines a function called `apply_transformation` that takes a matrix `A` and a vector `v`. In Python, a list is an ordered collection that can be changed. Here, we represent a 2D vector as a list of two numbers, like `[2, 3]`. A matrix is a list of lists: `A = [[2, -1], [1, 3]]` means the first row is `[2, -1]` and the second row is `[1, 3]`. The function computes the output by multiplying each row of the matrix by the vector, adding the products. For the first component, it multiplies the first row's first element by the first component of the vector, and the first row's second element by the second component, then adds them. The same is done for the second row. The function returns a new list with the two results. In the example, we use the same numbers as in the hand calculation, and the output is `[1, 11]`, matching what we got by hand.


## Common misconceptions

**A matrix is just a table of numbers** — A matrix is just a rectangular array of numbers, and multiplying is a rote procedure.

A matrix stores a linear transformation. Each column tells you where a basis vector goes. The familiar row-times-column multiplication is just the rule for applying that transformation to an input. In fact: A matrix is a compact record of a linear transformation, and matrix-vector multiplication is the act of applying that transformation to a vector.

**The origin can move** — A linear transformation can slide the whole space around, like moving a sheet of paper.

Sliding (translation) does not preserve the origin, and it does not satisfy the scaling rule: sliding a twice-stretched arrow is not the same as stretching the slid arrow. In fact: A linear transformation must fix the origin. A translation is a separate, affine transformation, which a neural network handles with a bias term.

**The transformation depends on the coordinate system** — Changing the basis changes what the transformation actually does to the points.

The rule that takes a point and moves it is independent of how we choose to label the points. Only the matrix representation changes, not the underlying map. In fact: The transformation is a geometric reality; the matrix is a coordinate-dependent description of it. This is why eigenvalues and eigenvectors matter — they are independent of the basis.

**A linear transformation can bend lines** — Any smooth stretching could bend straight lines into curves.

Linearity specifically forbids bending: the image of a straight line is always a straight line (or a point). This is what makes the transformation completely determined by the images of the basis vectors. In fact: A linear transformation preserves straightness and parallelism. It can rotate, scale, shear, or collapse, but never curve.

**Composing transformations is commutative** — If I apply transformation A then B, I get the same result as applying B then A.

The order in which you apply transformations matters. For example, a rotation followed by a reflection is usually different from the reflection followed by the rotation. This is why matrix multiplication is not commutative. In fact: Composition is order-dependent: AB means apply B first, then A. This order is what encodes a sequence of operations like a neural network's layers.


## In short

A linear transformation is a rule that moves every point in space while keeping straight lines straight, the origin fixed, and respecting addition and scaling. It is completely described by what it does to the basis vectors, and that description is its matrix. Matrix-vector multiplication is the act of applying the transformation. This idea is the foundation of all of machine learning: every layer of a neural network is a linear transformation followed by a nonlinearity, and the whole field of linear algebra exists to study these transformations.


## Where this is used

Linear transformations are everywhere in machine learning, often hidden inside a layer that also has a bias and a nonlinearity. In a neural network, a fully connected layer computes `output = activation(W @ input + b)`. The matrix `W` is a linear transformation: it rotates, scales, and shears the input space. The bias `b` shifts the result, and the activation (like ReLU) introduces nonlinearity. Without the nonlinearity, stacking layers would just be another linear transformation, and the network could not learn complex patterns. In large language models like GPT, each transformer block contains linear transformations in the attention mechanism (the query, key, and value projections) and in the feed-forward layers. These are matrix multiplications on the embedded tokens. When you use a library like PyTorch, you write `nn.Linear(in_features, out_features)`, which internally stores a weight matrix and a bias vector. The forward pass is `out = input @ weight.T + bias`. The weights are learned during training by adjusting them to reduce loss, and that adjustment is itself a linear transformation (the gradient). In retrieval-augmented generation (RAG), documents and queries are converted into embeddings — vectors that represent meaning. The similarity between a query and a document is often the dot product, which is a linear operation. The embeddings are often produced by a model that has linear transformations in its final layers. In agentic systems, a model that plans and calls tools may use a linear layer to map its internal state to a choice of action (tool call or output). In everyday applied machine learning, linear transformations appear in recommendation systems (matrix factorization), image processing (filters are convolutions, which are linear), forecasting (linear regression is a linear transformation), fraud detection (logistic regression), and speech recognition (the input features are transformed linearly before being fed to a recurrent or transformer network). A developer building a model writes code like `layer = nn.Linear(784, 256)` for a neural network, or uses `np.linalg.solve` to fit a linear regression. When done badly — for example, forgetting the bias term, or choosing an activation that is too linear — the model may fail to learn. Understanding linear transformations helps you debug shape mismatches and grasp why the order of operations matters.


---

Read this on the web: https://basavapurushottam.com/learn-ai/linear-transformations
