# Image Segmentation

Assigning a label to every pixel, delineating exact object boundaries rather than bounding boxes.

**Rests on:** Stride and Padding, Tensors and Shapes

**Leads to:** Liveness Detection

*Deep Learning · Computer Vision*


## Why learn this

Image segmentation allows a computer system to understand *what* is in an image down to the smallest detail, rather than just knowing that certain things are present. This precision is critical for tasks where edges matter. For example, in medical imaging, identifying the exact boundaries of a tumour can determine whether surgery is possible. In autonomous driving, knowing the precise edge of the road or the shape of an obstacle prevents accidents caused by vague estimations. In photo editing, it allows you to remove a background seamlessly without leaving jagged artifacts around a person. It turns a rough estimate into a usable, pixel-perfect map of reality.


## The intuition

When you look at a photograph, your eye instantly distinguishes between different parts of the scene. If there is a cat sitting on a sofa, your brain knows exactly which points belong to the cat and which belong to the sofa. Image segmentation does this digitally: it assigns a specific category to every single pixel in the image. 

Consider a photo with 10 million pixels. A simpler approach, called object detection, might draw a rough rectangle around the cat. This rectangle says, 'there is a cat somewhere here,' but it also includes parts of the sofa and the floor behind the cat. Segmentation goes further. It draws a precise outline that follows the curve of the cat’s whiskers, its tail, and its ears. Every pixel inside that outline is labeled 'cat'; every pixel outside is labeled 'sofa' or 'floor.' 

This level of detail comes with a cost. Creating training data for this system is much harder than for simple object detection. If you use the rectangle method, you only need to draw one box per cat. With segmentation, someone must manually label every single pixel in that box as 'cat' and every pixel outside as 'not cat.' In an image with millions of pixels, this is a time-consuming job.


## An analogy

Imagine a teacher taking attendance in a large lecture hall. A bounding-box approach (object detection) is like the teacher glancing at the room and saying, 'There are 50 students here,' while drawing a vague circle around the entire group of people. The teacher knows roughly where they are but doesn't know who is who or exactly where one person ends and another begins.

Image segmentation is like the teacher reading out every seat number in the hall. For seat 1A, they say 'occupied.' For seat 1B, they say 'empty.' They assign a specific status to every single individual spot. This allows the teacher to know exactly who is present and where they are sitting. 

The analogy breaks down in one key way: in a classroom, a person occupies a distinct, solid space. In an image, pixels can blend together. A single pixel might be on the edge of a blue shirt and a white collar, making it hard to assign it purely to one category without 'blurring' the line. The segmentation system must handle these soft edges carefully.


## Definition

Image segmentation is the process of labeling every individual pixel in an image with a specific category, such as 'road,' 'car,' or 'sky,' creating a precise map of objects and backgrounds rather than just detecting their rough locations.

Two common variations exist: one which labels all pixels of the same type together (e.g., all cars are labeled 'car' regardless of which car it is), and another which distinguishes between individual objects of the same type (e.g., labeling 'car #1,' 'car #2,' etc.), as well as combining both to identify every distinct object and its surrounding background in a single view.


## How it connects

You have previously learned that neural networks can classify entire images, deciding if a picture is generally of a cat or a dog. Segmentation extends this by applying classification to tiny regions within the image instead of the whole picture. This concept works closely with object detection, which finds rough locations (bounding boxes) but lacks pixel-level detail, and with instance segmentation, which adds the ability to distinguish between multiple individual objects of the same class, such as separating two cars parked next to each other.


## The mathematics

In image segmentation, we want to decide what each individual pixel in a picture is. Is it part of the road? Part of a car? Part of the sky?

The computer model does not guess directly. Instead, for every single pixel, it produces a list of raw numbers called **scores**. If there are three possible classes (road, car, sky), the model gives us three scores: one for 'road', one for 'car', and one for 'sky'.

These scores can be any number: positive, negative, or zero. They do not yet mean 'probability' because they do not add up to 1, and some might be negative. We need a way to turn these raw scores into probabilities that make sense together.

We use an operation called **softmax**. Think of softmax as a two-step process: first, it makes all the numbers positive; second, it scales them so they add up to exactly 1.

Here is how softmax works for one pixel:

Step 1: Take each raw score and raise the constant e (approximately 2.718) to the power of that score. This is called exponentiation. 
- Why? Because e raised to any number is always positive. This removes any negative scores.
- Also, this step magnifies differences. If one score is higher than another, raising them to powers makes the gap between them even larger.

Step 2: Add up all these new positive numbers. Call this total sum.

Step 3: Divide each of the exponentiated numbers by that total sum. 
- The result for each class is its probability.
- Because we divided every part by the whole, the probabilities will always add up to 1.

The final label for the pixel is simply the class with the highest probability.

Let us write this out clearly for one pixel with C classes. Let z_1, z_2, ..., z_C be the raw scores for the C classes. 

First, we calculate the exponentiated value for each class i:
   e^(z_i)

Next, we calculate the sum of all these exponentiated values:
   Sum = e^(z_1) + e^(z_2) + ... + e^(z_C)

Then, the probability p_i for class i is:
   p_i = e^(z_i) divided by Sum

Finally, we pick the class i where p_i is the largest.

Symbol glossary:
- z_i: The raw score given by the model for class i. It can be any real number (positive, negative, or zero).
- C: The total number of classes (e.g., 3 if we have road, car, sky).
- e: Euler's number, a constant approximately equal to 2.718.
- p_i: The probability that the pixel belongs to class i, after applying softmax.

Note: The model is trained by comparing these probabilities to the true answer using a formula called cross-entropy loss. However, for understanding how to read the output, you only need the softmax steps above.


## A worked example

Suppose we have a tiny image with just 2 pixels. The model needs to classify each pixel into one of three classes: road, car, or sky.

For Pixel 1, the model outputs these raw scores:
- Score for 'road' (z_1) = 2.0
- Score for 'car' (z_2) = 1.0
- Score for 'sky' (z_3) = 0.5

We apply softmax to find the probabilities.

Step 1: Exponentiate each score using e ≈ 2.718.
- For road: e^(2.0) ≈ 7.389
- For car: e^(1.0) ≈ 2.718
- For sky: e^(0.5) ≈ 1.648

Step 2: Calculate the sum of these exponentiated values.
- Sum = 7.389 + 2.718 + 1.648 = 11.755

Step 3: Divide each exponentiated value by the sum to get the probabilities.
- Probability for road (p_1) = 7.389 / 11.755 ≈ 0.629
- Probability for car (p_2) = 2.718 / 11.755 ≈ 0.231
- Probability for sky (p_3) = 1.648 / 11.755 ≈ 0.140

Check: 0.629 + 0.231 + 0.140 = 1.000. The probabilities sum to 1.

The highest probability is 0.629, which corresponds to 'road'. So, Pixel 1 is labeled as road.

Now for Pixel 2, the model outputs these raw scores:
- Score for 'road' (z_1) = 0.0
- Score for 'car' (z_2) = 3.0
- Score for 'sky' (z_3) = 1.0

Step 4: Exponentiate each score.
- For road: e^(0.0) = 1.0
- For car: e^(3.0) ≈ 20.086
- For sky: e^(1.0) ≈ 2.718

Step 5: Calculate the sum.
- Sum = 1.0 + 20.086 + 2.718 = 23.804

Step 6: Divide each exponentiated value by the sum.
- Probability for road (p_1) = 1.0 / 23.804 ≈ 0.042
- Probability for car (p_2) = 20.086 / 23.804 ≈ 0.844
- Probability for sky (p_3) = 2.718 / 23.804 ≈ 0.114

Check: 0.042 + 0.844 + 0.114 = 1.000.

The highest probability is 0.844, which corresponds to 'car'. So, Pixel 2 is labeled as car.

Result: The final segmentation mask assigns 'road' to Pixel 1 and 'car' to Pixel 2. The probabilities also tell us the model's confidence: it is very confident about Pixel 2 (0.844) but less certain about Pixel 1 (0.629).


## From scratch, in Python

```python
# We'll represent an image as a list of lists (each inner list is a row of pixels).
# We'll assume 3 classes: road, car, sky.

# Raw scores for a 2x1 image (2 pixels, 3 classes)
scores = [[2.0, 1.0, 0.5],  # pixel 0
          [0.0, 3.0, 1.0]]  # pixel 1

# Python building blocks introduced here:
# - for loop: repeats a block for each item in a list
# - sum(list): adds all numbers in a list
# - enumerate: gives both the index and the value

# Step 1: Compute softmax for each pixel
import math
def softmax(scores_one_pixel):
    """Convert a list of raw scores to probabilities that sum to 1."""
    # Exponentiate each score
    exp_scores = [math.exp(s) for s in scores_one_pixel]
    # Sum of exponentiated scores
    total = sum(exp_scores)
    # Divide each by the total
    probabilities = [e / total for e in exp_scores]
    return probabilities

# Step 2: Apply softmax to every pixel
probs = []
for pix in scores:
    probs.append(softmax(pix))

# Step 3: For each pixel, find the class with the highest probability
class_names = ['road', 'car', 'sky']
labels = []
for p in probs:
    max_index = 0
    for i in range(1, len(p)):
        if p[i] > p[max_index]:
            max_index = i
    labels.append(class_names[max_index])

print(probs)   # [[0.629, 0.231, 0.140], [0.042, 0.844, 0.114]]
print(labels)  # ['road', 'car']

```


## The same thing, with a framework

```python
import numpy as np

# Simulate a model output for a 2-pixel image with 3 classes
scores = np.array([[2.0, 1.0, 0.5],
                   [0.0, 3.0, 1.0]])

# numpy.exp applies e^x element-wise, and sum across classes (axis=1)
exp_scores = np.exp(scores)
probs = exp_scores / exp_scores.sum(axis=1, keepdims=True)

# Class with highest probability
labels = np.argmax(probs, axis=1)

print(probs)   # [[0.629, 0.231, 0.140], [0.042, 0.844, 0.114]]
print(labels)  # [0, 1]  -> class indices: 0=road, 1=car
```


## What the code does

The plain Python code first introduces `import math` (to get the exponential function), and a `function` called softmax that takes a one-pixel list and returns probabilities. The loop `for pix in scores` iterates over each pixel. Then we find the index of the maximum probability using a simple comparison loop. The NumPy version does the same thing but in fewer lines: `np.exp(scores)` exponentiates every element at once (that is exactly what the Python loop did), and the division is performed element-wise on the whole array. `np.argmax(probs, axis=1)` finds the index of the highest probability along each row — the same as our manual `for` loop.


## Common misconceptions

**Segmentation is just classification with more classes** — It's like the cat vs dog classifier, but we just have many more categories.

Classification gives one label for the whole image. Segmentation gives a label for every pixel, which means the model must preserve spatial information — a huge additional requirement. And the training data cost is drastically higher. In fact: Segmentation is pixel-level classification, but the architecture and loss functions are designed to keep spatial resolution, unlike global classifiers which discard it.

**A bounding box is good enough for segmentation** — If we have the box, we can just crop and label the middle pixel.

A box includes background and misses object parts that are outside the box. Cropping inside the box won't give the precise shape. In fact: Segmentation needs to know which exact pixels belong to the object, including concave sections — a rectangle cannot capture that.

**All pixels are independent** — The model labels each pixel by itself; it doesn't matter what the neighbours are.

A single pixel has no colour context; segmentation models (like fully convolutional networks) use local context through filters, and global context through downsampling/upsampling. In fact: Segmentation models are specifically designed to incorporate neighbouring pixel information, because that is what makes boundaries coherent.

**Softmax probabilities are exact confidences** — If the model says 0.9 for road, it is 90% sure it is road.

Probabilities are calibrated only if trained with a proper loss and sometimes need post-hoc calibration. The relative ordering matters more for the label. In fact: Softmax gives a relative ranking; the largest probability is usually the best guess, but the raw number is not a true confidence percentage.

**Segmentation is solved by any CNN** — A typical image-classification CNN can also do segmentation if we add more layers.

Classification CNNs downsample aggressively and lose spatial detail; segmentation needs upsampling to restore full resolution, so it uses architectures like U-Net or FCN. In fact: Segmentation requires specialised networks that preserve or reconstruct spatial information through skip connections and deconvolution.


## In short

Image segmentation labels every pixel of an image with a class, producing a precise mask. It is more expensive than classification because every pixel needs a label. The maths reduces to softmax (exponentiate, normalise to probabilities) applied to each pixel's scores. In code, you compute softmax per pixel and take the argmax. This unlocks precise downstream tasks like medical imaging, autonomous driving, and photo editing.


## Where this is used

Segmentation is everywhere in applied AI. In **large language models**, segmentation doesn't appear directly in the text generation process, but it is used in multimodal models that take images as input — e.g., GPT-4V can segment objects in an image to ground its descriptions. The mechanism is that an image encoder (often a vision transformer) produces patches that are processed alongside text tokens; a segmentation head (a convolutional or transformer layer) converts those features into a per-pixel mask. In **small language models on devices**, segmentation is used in camera apps (portrait mode) — a small segmentation network runs on the phone's NPU to separate the subject from the background. The model is distilled from a larger one to fit the memory budget, often using a lightweight U-Net architecture. In **retrieval-augmented generation**, segmentation is not typically used, but the idea of *masking* — selecting relevant parts of an input — is analogous; in some multimodal RAG systems, segmentation masks are used to focus the retriever on regions of an image that contain the queried object. In **agentic systems**, a robot or AR assistant that plans actions needs precise segments to know exactly which object to pick up. The agent calls a vision module that outputs masks, and then the plan is constructed with those masks as constraints. In **everyday applied work**, segmentation is used in medical imaging (MRI to delineate tumours), satellite imagery (to classify land use), and autonomous vehicles (to segment roads, lanes, pedestrians). A developer using PyTorch would write: `model = torchvision.models.segmentation.fcn_resnet50(pretrained=True)` then feed an image and take `output['out']` to get per-pixel logits, apply softmax to get probabilities, and `argmax` to get the label map. They also need to handle the output resolution — often smaller than the input, so they resize the mask back to the original image size. What breaks when done badly: if the model is trained on one camera's images and deployed on another (domain shift), the masks become noisy; if the training data has wrong labels at boundaries, the model learns to be blurry at edges — boundary accuracy is what distinguishes usable from impressive.


---

Read this on the web: https://basavapurushottam.com/learn-ai/image-segmentation
