# Activation Functions

The nonlinearity applied after each layer's linear transformation.

**Rests on:** Forward Propagation, Linear Transformations

**Leads to:** Kolmogorov-Arnold Networks, Layer Normalization, ReLU and Dead Neurons, Sigmoid and Tanh, Xavier Initialization

*Deep Learning · Deep Learning*


## Why learn this

This concept is the reason deep learning can learn anything complicated at all. Without activation functions, stacking layers of a neural network would be pointless — the whole stack would behave like a single straight-line formula, no matter how many layers you added. This unlocks your understanding of every modern AI system: large language models, image recognizers, and speech assistants all rely on the nonlinearity that activations provide. Master this and you can understand why networks are deep, why certain architectures work, and how to debug a model that isn't learning.


## The intuition

Think of a neural network as a series of translators. Each layer takes in information, processes it, and passes it to the next. If each translator simply multiplied their input by a number and added another number — a linear operation — then the whole team of translators would be equivalent to just one translator doing a single multiplication and addition. No matter how many layers you had, you couldn't represent anything more complex than a straight line. Activation functions break this limitation. They are like a translator who sometimes says 'I won't pass on anything negative' or 'I'll squeeze whatever I receive into a range between 0 and 1'. These nonlinear rules allow the network to represent curves, corners, and all the intricate patterns in data — like recognizing a face or understanding a sentence. Without them, deep learning would be just linear regression in disguise.


## An analogy

A library with a security guard. Imagine a library where each floor is a layer of neurons. Books flow through the floors — information. On each floor, a librarian (the linear transformation) rearranges the books according to a fixed rule (weights and bias). Without an activation function, the librarians would just pass the rearranged books on, and the final result after many floors would be the same as if only one librarian had done all the rearranging — you'd just get a scaled and shifted version of the original. Now add a security guard at the exit of each floor: the activation function. The guard decides which books are allowed to proceed to the next floor. For example, the guard might say, 'Only books with positive page counts can pass; anything negative is turned back.' Or 'I'll compress the page count to between 0 and 1, so even huge numbers become a fraction.' This rule is nonlinear — it breaks the chain of simple scaling. Now, if a book has a negative page count on one floor, it might be blocked, but on the next floor a different guard might allow negative numbers through. The combination of these guards across floors creates variety and complexity that a single librarian could never achieve. The point where the analogy stops: In the library, the guards are arbitrary and fixed, but in neural networks, the 'guards' have a specific mathematical form (like keeping only positive numbers), and they are fixed by the model designer, not learned. Also, the guards don't just block or pass; they sometimes transform the number (like compressing it), which is more than a binary decision.


## Definition

An activation function is a nonlinear mathematical rule applied to the output of each neuron (or layer) in a neural network, which determines whether and how that neuron's input influences the rest of the network, enabling the network to learn complex patterns.


## How it connects

This concept is the natural next step after understanding a single artificial neuron, which you have notes on. The neuron computes a weighted sum and adds a bias — that's the linear transformation. The activation function is applied right after that, before passing the result to the next layer. It is also closely tied to your notes on Deep Learning Basics: neurons, weights, and loss functions. Without activations, your neural network would be just a linear regression model, no matter how many layers you add. This is why activations are the cornerstone of deep learning.


## The mathematics

Activation functions are part of a branch of mathematics called nonlinear functions, which is a step beyond the simple arithmetic you know. A function is a rule that takes an input number and gives you an output number. A linear function is like a straight-line graph: if you double the input, the output doubles too (and adding inputs adds outputs). A nonlinear function is anything that doesn't follow that rule. For example, the rule 'if the input is negative, output 0; otherwise output the input' is nonlinear because if you input -1 you get 0, but if you input 1 you get 1 — that's not a constant scaling. Activation functions are specifically chosen to be nonlinear. They take the raw score from a neuron (a single number, like 1.6) and transform it. There are several common ones. The ReLU function is the simplest: it outputs the input if it's positive, and 0 otherwise. This is like a bouncer saying 'no negative vibes here.' The sigmoid function takes any number and squeezes it into a range between 0 and 1, like compressing a huge range of volumes into a whisper and a shout. The tanh function squeezes between -1 and 1. The choice of activation function has a huge impact on how well the network learns. The key property that makes them useful is their nonlinearity, which allows the network to represent curved boundaries between categories.

Let z be the raw score (the weighted sum plus bias). The three most common activation functions are:

- ReLU: f(z) = max(0, z)
- Sigmoid: f(z) = 1 / (1 + e^(-z))
- Tanh: f(z) = (e^z - e^(-z)) / (e^z + e^(-z))

Each of these is applied to the raw score z to produce the final output of that neuron.


## A worked example

```
You have a single neuron with one input. The weight is 2.0, the bias is -1.0. We'll apply three different activation functions to the same raw score to see how they change. Raw score z = 2.0 * 1.0 + (-1.0) = 1.0.

Step 1: z = 2.0 * 1.0 - 1.0 = 1.0   # First, compute the linear part: the weighted sum plus bias.
Step 2: ReLU(1.0) = max(0, 1.0) = 1.0   # ReLU keeps positive numbers unchanged.
Step 3: Sigmoid(1.0) = 1 / (1 + e^(-1.0)) = 1 / (1 + 0.3679) = 1 / 1.3679 ≈ 0.731   # Sigmoid squeezes the value into (0,1). e is a constant approximately 2.718.
Step 4: Tanh(1.0) = (e^(1.0) - e^(-1.0)) / (e^(1.0) + e^(-1.0)) = (2.718 - 0.368) / (2.718 + 0.368) = 2.350 / 3.086 ≈ 0.762   # Tanh squeezes into (-1,1).

Result: For the same raw score of 1.0, ReLU outputs 1.0, sigmoid outputs about 0.731, and tanh outputs about 0.762. Different activations change the scale and shape of the output, affecting how the network interprets the signal.
```


## From scratch, in Python

```python
import math

# Raw score calculation function
def neuron_output(inputs, weights, bias):
    # inputs is a list of numbers, weights is a list of numbers of the same length
    total = 0
    for i in range(len(inputs)):
        total = total + inputs[i] * weights[i]
    total = total + bias
    return total

# Activation functions
def relu(x):
    if x > 0:
        return x
    else:
        return 0

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def tanh(x):
    # we can use math.tanh, but here's the formula
    return (math.exp(x) - math.exp(-x)) / (math.exp(x) + math.exp(-x))

# Example usage
inputs = [1.0, 2.0, 3.0]
weights = [0.5, -0.2, 0.8]
bias = -0.1

z = neuron_output(inputs, weights, bias)
print('Raw score:', z)
print('ReLU output:', relu(z))
print('Sigmoid output:', sigmoid(z))
print('Tanh output:', tanh(z))
```


## The same thing, with a framework

```python
import numpy as np

# Use numpy for arrays
def relu_np(z):
    return np.maximum(0, z)  # element-wise max

def sigmoid_np(z):
    return 1 / (1 + np.exp(-z))

def tanh_np(z):
    return np.tanh(z)

# Example
z_vector = np.array([-1.0, 0.0, 2.0])
print('ReLU:', relu_np(z_vector))
print('Sigmoid:', sigmoid_np(z_vector))
print('Tanh:', tanh_np(z_vector))
```


## What the code does

We start by defining a function 'neuron_output'. A function is a named block of code that takes inputs (called parameters), does something with them, and can return a result. Here, the inputs are a list of numbers (inputs), a list of weights of the same length, and a single bias. A list is an ordered collection of items, like [1.0, 2.0, 3.0]. We use a 'for' loop to go through each position in the list: 'for i in range(len(inputs))' means 'for each index i from 0 to the number of inputs minus 1'. Inside the loop, we add the product of the input and weight to a running total. After the loop, we add the bias and return the total. Then we define three activation functions: 'relu' uses an 'if' statement to check if x is positive; 'sigmoid' uses the mathematical formula; 'tanh' uses another formula. Finally, we create example inputs and weights, compute the raw score, and print the results of applying each activation function.


## Common misconceptions

**Activation functions are optional** — I can build a neural network without activation functions; it will just be simpler.

Without a nonlinear activation, a multi-layer network reduces to a single linear transformation, so adding layers doesn't help. It cannot learn non-linear patterns like XOR. In fact: Activation functions are mandatory in hidden layers; they are what give neural networks their power to model complex relationships.

**ReLU is just a max; any max will do** — As long as it's nonlinear, it works the same.

Different activations have different gradient properties. Sigmoid saturates at both ends, causing vanishing gradients. ReLU has a constant gradient for positive inputs, which helps training deep networks. In fact: The choice of activation affects how easily the network learns. ReLU is preferred for hidden layers because it avoids saturation in the positive region.

**Sigmoid is always the right output** — For any classification problem, I should use sigmoid at the output.

Sigmoid is for binary classification or probabilities. For multi-class, softmax is used. For regression, a linear output is used. The output activation must match the task. In fact: The output layer's activation is dictated by the problem: sigmoid for binary, softmax for multi-class, linear for regression.

**Activation functions are learned** — The network learns the best activation function during training.

Activations are fixed, non-parametric functions chosen by the designer. There are learnable variants (like Swish via a learned parameter), but standard ReLU has no learnable parameters. In fact: Activation functions are typically predefined, not learned. They are applied without adjusting their behaviour during training.


## In short

Activation functions inject nonlinearity into neural networks, making deep learning powerful. Without them, a deep network would be equivalent to a single linear layer. Choose ReLU for hidden layers to avoid vanishing gradients; use sigmoid or tanh only when needed (e.g., output probabilities). The output layer's activation must match the task. Remember: gradient flow is the primary criterion — choose activations that keep gradients flowing through the network.


## Where this is used

Activation functions are used in every neural network, from large language models to small on-device models. Here are concrete examples:

1. **Large Language Models (like GPT)**: In transformers, the hidden layers use the GELU activation function (a smooth version of ReLU). The output layer uses a softmax to produce probabilities over words, but that is a different kind of activation. The key: without ReLU-like functions, the deep stack of layers would collapse into linear operations and the model could only learn linear relationships, which is far too weak for language.

2. **Small Models on Devices (TinyML)**: Models running on phones or microcontrollers often use ReLU because it is computationally cheap — it only involves a comparison, not exponentials. This saves battery and memory. Some use quantized versions of ReLU to further speed things up.

3. **Retrieval-Augmented Generation (RAG)**: RAG uses an embedding model to convert text into vectors. These embedding models are neural networks with activation functions in their hidden layers. The activations help the model learn to place similar texts close together in vector space, which enables the similarity search that RAG relies on.

4. **Agentic Systems**: Agents that plan and act often use reinforcement learning, where the policy network (a neural network) uses activations like tanh to output actions within a bounded range (e.g., -1 to 1 for controlling a robot).

5. **Everyday Applied ML**: In fraud detection, classifiers use sigmoid in the output layer to give a probability of fraud. In image recognition, convolutional networks use ReLU after each convolution to introduce nonlinearity, enabling them to detect edges, shapes, and objects.

What a developer writes: In PyTorch, a developer writes `torch.nn.ReLU()` or `torch.nn.GELU()` as a layer. Underneath, it's a simple operation applied to the output of the previous linear layer. If they do it badly — e.g., using sigmoid in a 100-layer network — the gradients vanish, training stalls, and the model never learns. This is why ReLU is the default for hidden layers.


---

Read this on the web: https://basavapurushottam.com/learn-ai/activation-functions
