# Sigmoid and Tanh

The classic saturating activations, squashing inputs into a bounded range.

**Rests on:** Activation Functions, Forward Propagation

*Deep Learning · Deep Learning*


## Why learn this

You use functions like these every time you need to convert a raw score into a meaningful percentage or a balanced judgment. Imagine you are building a system to decide if an email is spam. The computer calculates a 'confidence score' of 150. You cannot simply tell a human 'the confidence is 150'. You need a rule that turns that big number into something between 0% and 100%. Sigmoid does exactly this: it takes any input number and squeezes it tightly into the range 0 to 1, which you can interpret directly as a probability. Similarly, if you want a system that balances two opposing choices, like 'buy' vs 'sell', you might need values between -1 and 1. Tanh provides that balance. Understanding how these functions work is essential because they control how confidently a model expresses its answers and, critically, how much information flows back through the system to allow learning. Without this understanding, it is impossible to know why some models learn quickly while others stall.


## The intuition

Think of these functions as extreme squeezers or compressors. Take any number you can imagine: 1,000,000; -50; 0.0001; or even just 2. The Sigmoid function takes that input and transforms it into a result that never falls outside the narrow lane between 0 and 1. 

Let us look at specific numbers to see how this 'squeezing' behaves:

- If you put a large positive number like 3 into Sigmoid, the output is approximately 0.95. It is very close to the top of the lane (1).
- If you put exactly 0 in, the output is exactly 0.5. This is the midpoint.
- If you put a large negative number like -3 in, the output is approximately 0.05. It is very close to the bottom of the lane (0).

As the input numbers get larger in either direction (more positive or more negative), the output gets closer and closer to the edges (1 or 0) but never actually touches them. This behavior is called 'saturation'. The function becomes flat at the edges.

Tanh works on the same principle but uses a different lane, from -1 to +1. 
- Input 3 becomes roughly +0.995 (near the top).
- Input 0 becomes exactly 0 (the neutral middle).
- Input -3 becomes roughly -0.995 (near the bottom).

The key feature here is that while Tanh centers the output at zero, it still saturates. When the function saturates, its slope becomes almost flat. Imagine a hill that is very steep in the middle but becomes completely flat on the sides. If you try to push a ball down that hill near the edge, it hardly moves. In learning systems, this 'flatness' means the system stops receiving signals about how to adjust itself, which can cause the entire process to stop learning effectively.


## An analogy

Imagine a master key that fits into almost any lock, but the lock has a physical stop that prevents the key from turning more than 360 degrees. If you have a key with a long shaft (a huge input number) and you try to turn it wildly in either direction, the tumblers inside the lock will hit their stops. Whether the input is 1 or 1,000, the lock's final position might be nearly identical because it has reached its mechanical limit. This is what happens when the function saturates: small changes in your input are lost because the output has nowhere left to go. Tanh is like a lock that starts in the neutral 'off' position and can turn fully clockwise or counter-clockwise, but still hits hard stops at either extreme.


## Definition

Sigmoid and Tanh are activation functions that take any real number as input and map it to a fixed, bounded range: Sigmoid outputs values strictly between 0 and 1, while Tanh outputs values strictly between -1 and 1, effectively 'squashing' wide ranges of input into narrow intervals.


## How it connects

These are the classic examples of activation functions, which are the mathematical tools that introduce nonlinearity into a neural network. Without these nonlinear steps, multiple layers of simple linear calculations would cancel each other out, making depth useless. Sigmoid and Tanh were the original choices for this role before simpler alternatives like ReLU became standard for hidden layers due to speed. They also illustrate the 'vanishing gradient' problem, where the signal used to update the model's memory fades away as it travels backward through many layers because the function's slope becomes too flat.


## The mathematics

The branch of mathematics we need here is 'functions' — a machine that takes an input number and produces an output number. You may have met this in school: something like 'f(x) = x + 2' that takes a number, adds 2, and gives you a result. That's it. We'll now build two specific functions. First, Sigmoid. It works in two steps: 1) take your input number, multiply it by −1 (change its sign), and then raise a special constant called 'e' (which is about 2.718) to the power of that result. 2) Add 1 to that answer, then divide 1 by that sum. The result is always between 0 and 1. Here's why: if your input is very big and positive, e^(−input) is very small (like 0.0001), so you get 1 divided by (1 + tiny), which is close to 1. If your input is very negative, e^(−input) is huge, so you get 1 divided by (1 + huge), which is close to 0. At input 0, you get 1 divided by (1+1) = 0.5. Tanh is very similar, but its output range is −1 to 1. Instead of doing it in two steps, you can think of it as: take the Sigmoid of your input, multiply by 2, and subtract 1. Or you can remember the formula: (e^input − e^(−input)) / (e^input + e^(−input)). At input 0, that gives 0; at plus infinity it gives 1; at minus infinity it gives −1.

Sigmoid(x) = 1 / (1 + e^(−x))
Tanh(x) = (e^x − e^(−x)) / (e^x + e^(−x)) = 2 * Sigmoid(2x) − 1


## A worked example

```
We'll compute the Sigmoid of 2, and the Tanh of 2, by hand. Note: we use the approximate value of e ≈ 2.718 for our step-by-step, but the actual answer uses the true value.

Step 1: e^2 ≈ 2.718^2 ≈ 7.389   # First we need e raised to the power of our input, 2. We'll keep this for later.
Step 2: e^(−2) ≈ 2.718^(−2) ≈ 0.1353   # Also compute e raised to the power of −2. This is just 1 divided by the previous result.
Step 3: Sigmoid(2) = 1 / (1 + e^(−2)) ≈ 1 / (1 + 0.1353) ≈ 0.8808   # Sigmoid of 2 is about 0.88 — a confident 'yes'.
Step 4: Tanh(2) = (e^2 − e^(−2)) / (e^2 + e^(−2)) ≈ (7.389 − 0.1353) / (7.389 + 0.1353) ≈ 0.9640   # Tanh of 2 is about 0.96 — even more skewed toward the positive end, because its range is −1 to 1.
Step 5: Check: Tanh(x) = 2 * Sigmoid(2x) − 1   # This is a helpful identity to remember: Tanh is a scaled and shifted Sigmoid.

Result: Sigmoid(2) ≈ 0.88, Tanh(2) ≈ 0.96. Both are large positive values because 2 is positive and far from 0.
```


## From scratch, in Python

```python
import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def tanh(x):
    return (math.exp(x) - math.exp(-x)) / (math.exp(x) + math.exp(-x))

# Test with some numbers
for val in [-3, -1, 0, 1, 3]:
    print(f"sigmoid({val}) = {sigmoid(val):.4f}, tanh({val}) = {tanh(val):.4f}")
```


## What the code does

We use the `math` library, which gives us `math.exp` to compute e raised to a power. The `def` keyword creates a function called `sigmoid` (or `tanh`) that takes one number `x`. Inside, we just write the formula we hand-computed above. The `for` loop tests our function with a few example inputs, printing the results rounded to four decimal places. This is the same as writing the arithmetic line by line, but now we can use it with any input.


## Common misconceptions

**Sigmoid output is a 'real' probability** — If sigmoid outputs 0.7, that's exactly the probability that the event happens.

Sigmoid outputs a number in (0,1), but it is only a calibrated probability if the model is trained with a proper loss function and temperature scaling. Neural networks are often overconfident. In fact: Sigmoid output is a confidence-like score between 0 and 1, but it should be interpreted as a probability only after calibration.

**Tanh is completely different from sigmoid** — Sigmoid and tanh are two unrelated functions.

They share the same exponential form, and there is a direct algebraic relationship: tanh(x) = 2*sigmoid(2x) − 1. In fact: Tanh is a scaled and shifted sigmoid: it's the same squashing shape but centered at 0 and ranging from −1 to 1.

**Sigmoid and tanh are used everywhere in modern nets** — All neural networks today use sigmoid or tanh as activations.

Modern deep networks overwhelmingly use ReLU or its variants in hidden layers, reserving sigmoid for output layers and gates. In fact: Sigmoid and tanh are used in specific spots — output layers for probabilities, and in gating mechanisms — but not as the default hidden-layer activation.

**Saturation is harmless** — It's fine if the output is close to 1 or 0, the model still works.

When saturated, the derivative becomes almost zero, so gradient-based learning grinds to a halt — the 'vanishing gradient' problem. In fact: Saturation is the key weakness of these functions; it's why they were abandoned for hidden layers in deep networks.


## In short

Sigmoid and tanh are the classic saturating activations. Sigmoid maps all inputs to (0,1) — a probability-like range — and tanh maps to (−1,1) by being a scaled and shifted sigmoid. They are still used in output layers and gates, but they cause vanishing gradients in deep hidden layers, which is why ReLU took over as the default. Remember: tanh(x) = 2*sigmoid(2x) − 1.


## Where this is used

These two activations appear throughout AI, though they've been replaced in many hidden layers by ReLU. Let's go through each domain: **Large language models**: Inside each 'attention' head of a transformer, there are 'gates' that decide how much information to let through — these often use a sigmoid. Also, the final output layer of an LLM uses a softmax, which is a generalised sigmoid for many classes. If you've ever seen GPT's 'logits' being converted to a probability, that's a sigmoid-style squashing. TinyML / small models on devices: When running a keyword spotter on a microcontroller, the final layer usually uses a sigmoid to decide 'wake word detected' vs 'not detected', because it outputs a probability between 0 and 1 that can be compared against a threshold. **Retrieval-augmented generation (RAG)**: When you retrieve the most relevant documents, you often compute the similarity between a query and each document. That similarity score might range from −1 to 1 (if you use cosine similarity), which you might then squash with a sigmoid to get a 'relevance probability' to decide whether to include a document in the context. **Agentic systems**: Agents that plan and call tools often use a confidence score based on a sigmoid output of a binary classifier (e.g., 'should I call the weather tool?'). The agent can then set a threshold — say 0.7 — above which it calls the tool, below which it asks the user. **Everyday applied ML**: In fraud detection, the model outputs a probability that a transaction is fraudulent — that's a sigmoid on the last layer. In recommendation systems, a sigmoid can predict the probability of a user clicking a recommended item. In computer vision, a bounding box regressor might use tanh to predict the center offset of an object, since it is bounded. In practice, a developer writes something like `import torch; outputs = torch.sigmoid(logits)` to turn raw scores into probabilities, or `tanh` to get values in a bounded range. If done badly — for example, using sigmoid in a deep hidden layer — you get vanishing gradients and the model stops learning. That's why we teach ReLU next.


---

Read this on the web: https://basavapurushottam.com/learn-ai/sigmoid-and-tanh
