# Tensors and Shapes

Multi-dimensional arrays and the discipline of tracking their dimensions through a computation.

**Rests on:** Vector Spaces and Rank

**Leads to:** Computational Graphs, Convolution Arithmetic, Crop Disease Detection, Dimensionality Reduction and PCA, Image Segmentation, Information Geometry, Keras and TensorFlow, Matrix Calculus and Jacobians, Matrix Multiplication, Multi-Head Attention, NumPy Broadcasting, Tall Array Computation

*Mathematical Foundations · Linear Algebra*


## Why learn this

If you have ever tried to combine two sets of data—like matching a list of customer names with their order histories—or built a model that predicts outcomes from multiple inputs, you have encountered the problem this concept solves. Modern tools for pattern recognition and prediction rely on organizing numbers into structured blocks called tensors. Understanding what a tensor is, and specifically how its dimensions (its shape) behave, is the single most important skill for fixing errors in these systems. When a model fails because the data structures do not align, knowing exactly why that happened allows you to fix it quickly. This page gives you the foundation to understand those alignment issues and prevents the confusion that comes when you later encounter complex models like neural networks or language processors.


## The intuition

You already use the idea of multi-dimensional organization in your daily work. Think of a spreadsheet. A single sheet with rows and columns is a table, which mathematicians call a 2-dimensional array. Now imagine you have a folder full of these spreadsheets—perhaps one for each month of the year. If you look at all those sheets together as one unit, you have created a 3-dimensional array: a stack of tables. A tensor is simply the name for any collection of numbers arranged in this way, regardless of how many layers it has. It can be 1-dimensional (a simple list), 2-dimensional (a table), 3-dimensional (a stack of tables), or even more layers deep.

What matters most is the shape. The shape is a list of numbers that tells you exactly how many items exist in each layer. For example, a spreadsheet with 10 rows and 5 columns has the shape (10, 5). If you take 20 of those identical spreadsheets and stack them, the resulting 3-dimensional tensor has the shape (20, 10, 5). This shape is not just a label; it is a strict promise about the structure of your data. Every time you perform an operation on this tensor—whether you are adding two tensors together or processing them through a model—the result must follow specific rules dictated by these shapes. Tracking the shape is like checking that two puzzle pieces fit before you try to push them together. You need to know which dimension represents what (e.g., time, category, or sample) to ensure the operation makes sense.


## An analogy

Think of a warehouse organizing crates of identical products. A single crate is a box with length and width. To store many crates, you stack them on shelves; a shelf is a row of these crates. To organize many shelves, you line them up in aisles. This creates a 3-dimensional structure: the aisle represents the first dimension, the shelf the second, and the position within the shelf the third. When you take a crate from one location and move it to another, you must know the dimensions of the container to ensure it fits. Similarly, when you combine data in computing, the shapes must align perfectly, or the operation cannot proceed. The analogy breaks down only when tensors have more than three dimensions—computers can handle four, five, or even fifty dimensions easily, whereas we cannot physically visualize them. However, the core idea remains the same: higher dimensions are just nested layers of lists, each layer having its own size.


## Definition

A tensor is a multi-dimensional array of numbers, and its shape is the sequence of integers that specifies how many elements exist along each dimension.


## How it connects

You know that in everyday life, a list of numbers is just a list, and a table of numbers is a table. A tensor generalizes this: it is a container that holds numbers in any number of dimensions, allowing you to treat a simple list, a table, and a stack of tables under one unified concept. This serves as the entry point into linear algebra, which is the branch of mathematics concerned with vectors (lists) and matrices (tables), and how they interact. In artificial intelligence, all data—whether a batch of images, a sequence of words, or a table of features—is organized as tensors. The shape of a tensor acts like the dimensions of a physical box, telling you exactly how many items fit in each direction. This understanding is essential for later topics involving the combination of these structures, where specific rules about their shapes determine whether operations are mathematically valid.


## The mathematics

Tensors are a general name for collections of numbers arranged in boxes of different shapes. You already know two specific shapes: a list of numbers (which mathematicians call a vector) and a table of numbers with rows and columns (which mathematicians call a matrix). A tensor is just the word we use when we want to talk about lists, tables, or even stacks of tables without having to invent new names for each one.

Think of the "shape" of a tensor as the sizes of its dimensions. We write the shape as a list of numbers in parentheses. For example, (2, 3) means there are two dimensions: the first has size 2, and the second has size 3.

A 1D tensor is a single list. Its shape is just one number, the length of the list. If the list has five numbers, we say its shape is (5).

A 2D tensor is a table. It has two dimensions: rows and columns. We write its shape as (number of rows, number of columns). For example, a table with 2 rows and 3 columns has shape (2, 3). This means there are 2 groups, and each group contains 3 numbers.

A 3D tensor is a stack of tables. It has three dimensions. We write its shape as (number of tables, number of rows per table, number of columns per table). For example, if you have 4 tables, and each table has 2 rows and 3 columns, the shape is (4, 2, 3).

The most important operation for tensors that share the same shape is element-wise addition. This means you take two tensors of the exact same shape and add the number in the first position of the first tensor to the number in the first position of the second tensor, then do the same for the second position, and so on. The result is a new tensor with that same shape.

There is also a special rule called broadcasting. Broadcasting allows you to add a single number (called a scalar) to every element of a tensor. It works as if you had created a huge tensor filled with that single number, matching the shape of the original tensor, and then added them together. You do not need to create the huge tensor; the computer just adds the single number to each spot.

Let us look at how we describe these shapes using real numbers.

Example 1: A 2D Tensor (Matrix)
Suppose we have a table of numbers called A. It has 2 rows and 3 columns.
We write its shape as (2, 3).
The first number in the shape, 2, tells us there are 2 groups of data.
The second number in the shape, 3, tells us each group has 3 items.

Example 2: A 3D Tensor
Suppose we have a stack of tables called B. It has 4 tables. Each table has 2 rows and 3 columns.
We write its shape as (4, 2, 3).
The first number, 4, is the count of tables.
The second number, 2, is the row count for each table.
The third number, 3, is the column count for each table.

Example 3: Element-wise Addition
Let us add two tensors, A and B, that both have shape (2, 3).
A = [[1, 2, 3], [4, 5, 6]]
B = [[10, 20, 30], [40, 50, 60]]

To find A + B, we add the numbers in the same positions:
Position (row 1, column 1): 1 + 10 = 11
Position (row 1, column 2): 2 + 20 = 22
Position (row 1, column 3): 3 + 30 = 33
Position (row 2, column 1): 4 + 40 = 44
Position (row 2, column 2): 5 + 50 = 55
Position (row 2, column 3): 6 + 60 = 66

The result is a new tensor of shape (2, 3):
[[11, 22, 33], [44, 55, 66]]

Example 4: Broadcasting with a Scalar
Let us add the single number 100 to every element of A from Example 3.
A = [[1, 2, 3], [4, 5, 6]]
Scalar c = 100

We add 100 to each number individually:
1 + 100 = 101
2 + 100 = 102
3 + 100 = 103
4 + 100 = 104
5 + 100 = 105
6 + 100 = 106

The result is:
[[101, 102, 103], [104, 105, 106]]
The shape remains (2, 3). The structure of the data has not changed, only the values.


## A worked example

We have a 2x3 tensor (2 rows, 3 columns) of numbers: [[1, 2, 3], [4, 5, 6]]. We want to add the scalar 10 to every element.

Step 1: Identify the shape. The shape is (2, 3). This means there are 2 rows and 3 columns.

Step 2: Take the first number in the tensor, which is 1. Add 10 to it.
1 + 10 = 11.

Step 3: Take the second number in the tensor, which is 2. Add 10 to it.
2 + 10 = 12.

Step 4: Take the third number in the tensor, which is 3. Add 10 to it.
3 + 10 = 13.

Step 5: Move to the second row. Take the fourth number, which is 4. Add 10 to it.
4 + 10 = 14.

Step 6: Take the fifth number, which is 5. Add 10 to it.
5 + 10 = 15.

Step 7: Take the sixth number, which is 6. Add 10 to it.
6 + 10 = 16.

Result: The resulting tensor is [[11, 12, 13], [14, 15, 16]]. The shape (2, 3) stayed the same — we just changed each number.


## From scratch, in Python

We'll use plain Python lists to represent a tensor. Here's how to create a 2D tensor and do element-wise addition.

```python
# Create a 2D tensor as a list of lists
tensor_a = [
    [1, 2, 3],
    [4, 5, 6]
]

# Create another 2D tensor of the same shape
tensor_b = [
    [10, 20, 30],
    [40, 50, 60]
]

# Get the shape manually
rows = len(tensor_a)          # 2
columns = len(tensor_a[0])    # 3

# Element-wise addition
tensor_c = []
for i in range(rows):
    row = []
    for j in range(columns):
        row.append(tensor_a[i][j] + tensor_b[i][j])
    tensor_c.append(row)

print(tensor_c)  # [[11, 22, 33], [44, 55, 66]]
```


## The same thing, with a framework

In NumPy, this is much simpler:

```python
import numpy as np

tensor_a = np.array([[1, 2, 3], [4, 5, 6]])
tensor_b = np.array([[10, 20, 30], [40, 50, 60]])

tensor_c = tensor_a + tensor_b

print(tensor_c)
print(tensor_c.shape)  # (2, 3)
```

NumPy's `+` operator does the loop for us, and it checks that the shapes match automatically. The `.shape` attribute tells us the dimensions.


## What the code does

We first define `tensor_a` as a list of two lists. Each inner list is a row. We use `len(tensor_a)` to get the number of rows, and `len(tensor_a[0])` to get the number of columns in the first row. We then loop over each row index `i` and each column index `j` to access each element, add the corresponding elements from `tensor_a` and `tensor_b`, and build up `tensor_c` row by row. This is the manual way to do element-wise addition. It's verbose but shows exactly what's happening.


## Common misconceptions

**Tensor shape is optional metadata** — The shape is just a nice label, but I can add or multiply any tensors regardless of shape.

Tensor operations are defined for specific shape combinations. Adding two tensors of different shapes gives an error unless broadcasting rules allow it. The framework enforces shape compatibility. In fact: The shape is fundamental to what the tensor is and what operations are valid on it.

**Reshape and transpose do the same thing** — Both reshape and transpose just change the dimensions, so I can use them interchangeably.

Reshape keeps the order of elements, just grouping them differently. Transpose swaps axes, which changes the logical layout of the data. They are not the same. In fact: Reshape is like rearranging blocks in a grid; transpose is like turning the grid sideways.

**A tensor must always be 2D** — I only need tables, so tensors are always tables (2D). I'll never see 3D or more.

Real data often has more than two dimensions: a color image is (height, width, channels) = 3D, a batch of images is 4D. In fact: Tensors can have many dimensions; the shape tells you how many.

**Scalar is the same as a tensor of shape (1,)** — A scalar (a single number) is just a tensor with one element, so shape (1,) is the same thing.

A scalar has zero dimensions; its shape is (). A 1D tensor with one element has shape (1,). They behave differently in broadcasting and multiplication. In fact: A scalar is just a number; a rank-1 tensor is a list with one item.


## In short

A tensor is a multi-dimensional array of numbers. Its shape is a tuple that tells you how many values are in each dimension. Every tensor operation checks shape compatibility. Reshape changes the grouping of elements without changing order; transpose swaps axes. Mastering shape tracking is the core debugging skill in AI programming.


## Where this is used

Tensors and their shapes are the foundation of every AI system. In **large language models** like GPT, a sentence is converted into a 2D tensor of shape (sequence_length, embedding_dimension). Each word becomes a list of numbers (its embedding), and the whole sentence is a table. When the model processes this, every operation — the multi-head attention, the feed-forward layers — is a tensor operation. The model's internal parameters are also tensors; for example, the weight matrix in a linear layer has shape (input_features, output_features). The shape mismatch is the classic error: if you try to multiply a (batch_size, sequence_length, input_features) tensor by a (input_features, output_features) weight, it only works if those last dimensions match. 

In **small language models** running on a phone, tensors are smaller but the same principles apply. The model is quantized — the tensors use 8-bit integers instead of 32-bit floats — which reduces memory and makes the shapes even more critical, because you need to convert shapes and types carefully. In **retrieval-augmented generation** (RAG), a query is converted into an embedding tensor, and it must be compared to a database of embeddings. The shape of the query embedding must match the shape of the document embeddings, otherwise the similarity computation fails. 

In **agentic systems** — AI agents that plan and call tools — tensors are used to represent state and action sequences. For example, a reinforcement learning agent might represent its observation as a tensor of shape (height, width, channels) for an image, and the policy network outputs a tensor of shape (num_actions,) representing probabilities of each action. Shape bugs surface as runtime errors. 

In everyday **applied machine learning** — recommendation, vision, forecasting, fraud detection — data is loaded as tensors. A tabular dataset becomes a 2D tensor of shape (num_samples, num_features). When you split it into batches, you create tensors of shape (batch_size, num_features). When you feed it to a neural network, the network's layers transform these tensors. 

A developer in practice writes code like `model(input_tensor)` and relies on the framework to check shapes. But when they get an error like 'MatMul: dimension mismatch', they need to debug by printing `input_tensor.shape` and checking the model's expected input shape. The mistake people make is confusing **reshape** (which just rearranges the same numbers into a new shape, as long as the total number of elements stays the same) with **transpose** (which flips axes, e.g., swapping rows and columns). NumPy's `reshape` and `transpose` are both cheap, but confusing them is a classic bug.


---

Read this on the web: https://basavapurushottam.com/learn-ai/tensors-and-shapes
