# Logistic Regression

Modelling the log-odds of a binary outcome as a linear function of predictors, giving calibrated probabilities.

**Rests on:** Cross-Entropy Loss, Linear Regression

**Leads to:** Bradley-Terry Model, Churn Prediction, Loss Function Selection, Rare Event Prediction

*Machine Learning · Machine Learning Foundations*


## Why learn this

Imagine you are designing a system to decide whether to approve a loan application. A simple 'yes' or 'no' answer is often not enough. You might need to know the *chance* of default to weigh it against other factors, or to report risk levels to regulators. Logistic regression is the standard tool for turning raw data—like income, credit history, and employment length—into a clear probability between 0% and 100%. It is the foundational building block for more complex AI systems, but its immediate value is giving you a quantifiable measure of confidence rather than just a binary guess. If you want to build systems that predict categories, such as detecting fraud, diagnosing disease, or predicting customer churn, this technique is your starting point.


## The intuition

Suppose you want to predict whether a specific email is spam (yes/no) based on two pieces of information: the number of exclamation marks in the subject line and whether the word 'winner' appears in the body. Let's say there are 3 exclamation marks and the word 'winner' is present. Logistic regression works by assigning a 'weight' to each piece of evidence that reflects how strongly it predicts spam. For example, let's assume the weight for 'winner' is very high (say, 5.0) because its presence is a strong indicator, while the weight for exclamation marks is lower (say, 0.5). It also has a 'baseline' setting (called bias) that starts the score at a neutral point, say -2.0. 

First, it multiplies each input by its weight: the 'winner' feature contributes 1 * 5.0 = 5.0 points, and the exclamation marks contribute 3 * 0.5 = 1.5 points. Then, it adds these products together along with the baseline: -2.0 + 5.0 + 1.5 = 4.5. This total, 4.5, is a raw score that can be any number from negative infinity to positive infinity. However, probabilities must be between 0 and 1. So, this score is passed through a special 'squashing' curve (the logistic function). This curve squeezes the infinite range of scores down into the 0-to-1 range. A high positive score like 4.5 gets squashed to a probability very close to 1.0 (99% chance of spam), while a low negative score would be squashed to near 0.0. The model 'learns' by looking at thousands of past emails, adjusting the weights and bias until its squashed probabilities match the actual labels (spam/not spam) as closely as possible.


## An analogy

Think of a judge deciding if a defendant should be released on bail. The judge looks at several risk factors: prior criminal record, current employment status, and residence stability. Each factor has a certain 'danger level' associated with it based on the judge's experience—the prior record might add significant risk points, while stable housing subtracts them. The judge mentally sums these weighted concerns to get a total risk score. However, instead of issuing a raw number like '73', the judge translates that sum into a practical judgment: 'the risk is about 85%'. This translation from a raw sum of concerns to a final probability is the core mechanism of logistic regression. The judge also has a personal baseline tendency to be either lenient or strict, which acts as the bias term. Just like the judge refines their judgment by reviewing past case outcomes, the model adjusts its weights through training on historical data. Caveat: This analogy simplifies reality; judges often consider complex interactions between factors that a simple linear sum might miss, whereas logistic regression strictly treats features as additive contributions to the risk score.


## Definition

Logistic regression predicts the probability of a binary outcome by computing a weighted sum of input features, adding a baseline value, and then passing that total through a sigmoid function to squash it into a range between 0 and 1.


## How it connects

This technique sits directly alongside Linear Regression, which you have likely encountered or will encounter soon; while linear regression predicts a continuous number by fitting a straight line through data points, logistic regression takes that same linear approach but applies a squashing curve to predict a probability instead. It is also the simplest form of a classifier, which is any model designed to assign data to a specific category, serving as the fundamental baseline against which more complex categorization methods are compared.


## The mathematics

Logistic regression is a method for predicting a probability. It works by taking several input numbers, combining them with specific weights to create a single number called a 'logit', and then transforming that logit into a value between 0 and 1 using the logistic function.

The process has two distinct stages.

Stage 1: Calculating the Logit (z)
The first stage is a linear combination. You take each input variable, multiply it by a corresponding weight, sum all those products together, and add a constant called the bias (or intercept). This result is called the logit, denoted here as z.

The formula for the logit is:
z = w1 * x1 + w2 * x2 + ... + wn * xn + b

Here is what each symbol means:
- x1, x2, ..., xn are the input values (the features). For example, 'hours studied' or 'hours slept'.
- w1, w2, ..., wn are the weights. These are numbers learned from data that determine how much influence each input has on the final result.
- b is the bias. This is a constant offset that shifts the decision boundary.
- z is the logit. It can be any real number, positive or negative.

Stage 2: Converting the Logit to a Probability
The second stage uses the logistic function (also called the sigmoid function) to squash the logit z into a probability p. The logistic function ensures the output is always between 0 and 1.

The formula for the probability is:
p = 1 / (1 + e^(-z))

Here is what each part means:
- p is the predicted probability that the outcome is 'yes' (or class 1).
- e is Euler's number, a mathematical constant approximately equal to 2.71828.
- (-z) is the negative of the logit calculated in Stage 1.
- e^(-z) means raising the number e to the power of -z. This is an exponential operation.
- The denominator (1 + e^(-z)) adds 1 to that exponential value.
- Finally, 1 divided by that denominator gives the probability.

Why this works: As z becomes very large and positive, e^(-z) becomes very close to 0, so p approaches 1/1 = 1. As z becomes very large and negative, e^(-z) becomes very large, so p approaches 1/large number ≈ 0. This creates the characteristic S-shaped curve of logistic regression.

Note on Notation: In some texts, you may see the formula written as p = e^z / (1 + e^z). These two formulas are mathematically equivalent. We use the form with e^(-z) here because it is numerically stable and easier to compute step-by-step when z is positive.


## A worked example

Let us calculate the probability that a student passes an exam based on two inputs: hours studied (x1) and hours slept (x2).

Given values:
- Weight for hours studied, w1 = 0.5
- Weight for hours slept, w2 = 0.2
- Bias, b = -1.0
- Input: hours studied, x1 = 5
- Input: hours slept, x2 = 7

Step 1: Multiply the first input by its weight.
Calculation: 0.5 * 5 = 2.5
Explanation: The contribution of 'hours studied' to the logit is 2.5.

Step 2: Multiply the second input by its weight.
Calculation: 0.2 * 7 = 1.4
Explanation: The contribution of 'hours slept' to the logit is 1.4.

Step 3: Sum the weighted inputs.
Calculation: 2.5 + 1.4 = 3.9
Explanation: We add the contributions together before adding the bias.

Step 4: Add the bias to get the logit z.
Calculation: 3.9 + (-1.0) = 2.9
Explanation: The raw score or logit is z = 2.9.

Step 5: Calculate the exponential term e^(-z).
Calculation: e^(-2.9)
Using the value of e ≈ 2.71828:
e^(-2.9) ≈ 0.055023
Explanation: We raise Euler's number to the power of negative 2.9. This value is small because the exponent is negative.

Step 6: Calculate the denominator (1 + e^(-z)).
Calculation: 1 + 0.055023 = 1.055023
Explanation: We add 1 to the exponential term.

Step 7: Divide 1 by the denominator to get the probability p.
Calculation: 1 / 1.055023 ≈ 0.9478
Explanation: The predicted probability of passing is approximately 0.948, or 94.8%.

Result: The model predicts a 94.8% chance that the student will pass the exam.


## From scratch, in Python

```python
import math

# Inputs: hours studied and hours slept
x1 = 5
x2 = 7

# Weights and bias (learned from data)
w1 = 0.5
w2 = 0.2
b = -1.0

# Compute the raw score
z = w1 * x1 + w2 * x2 + b

# Compute the probability using the logistic function
probability = 1 / (1 + math.exp(-z))

print('Raw score (z):', z)
print('Predicted probability:', probability)
```


## What the code does

We start by giving values to our inputs (x1 and x2), weights (w1 and w2), and bias (b). The line 'z = w1 * x1 + w2 * x2 + b' does the raw score calculation: multiply each input by its weight, add those products, and add the bias. Then we use Python's math library to get the exponential function, and compute the probability with the formula. The 'print' lines show the results. There is no special framework needed here—this is just arithmetic. A real library like scikit-learn would do this for you with one call, but seeing the steps helps you understand what happens under the hood.


## Common misconceptions

**Logistic regression is a classification algorithm, not a regression.** — I thought logistic regression is called regression because it predicts a continuous number, like linear regression.

Yes, the name is confusing, but it actually predicts a probability, which is continuous between 0 and 1, not a binary label. It is used for classification by applying a threshold (e.g., 0.5) to the probability. In fact: Logistic regression is a regression on the log-odds, then uses the logistic function to produce a probability. The output is a probability, and you decide the class by comparing that probability to a threshold.

**The sigmoid function is the only way to get probabilities.** — The sigmoid function is the only way to map a score to a probability.

There are other functions, like the probit function or the softmax (for multiple classes), that also map scores to probabilities. But sigmoid is the standard for binary classification because it is differentiable and has nice properties. In fact: The sigmoid is one of many possible 'squashing' functions, but it is the most common for binary classification. Softmax is used for multi-class problems, and other link functions exist in statistics.

**Coefficients are not directly probabilities.** — If my coefficient for income is 0.5, then a one-unit increase in income raises the probability of approval by 50 percentage points.

The coefficient affects the log-odds, not the probability directly. A one-unit change in income multiplies the odds by e^(0.5), which is about 1.65, but the effect on the probability depends on the current probability level. In fact: The coefficient tells you how the log-odds change, which translates to an odds ratio, not a simple probability increase. To interpret, you exponentiate the coefficient to get the odds ratio.

**Logistic regression can only draw a linear decision boundary.** — I thought logistic regression can learn any boundary, even curvy ones, because it is a powerful classifier.

Without adding polynomial features or other transformations, the decision boundary is always a straight line (or a flat hyperplane) in the input space. It cannot capture nonlinear patterns unless you engineer the features. In fact: By itself, logistic regression separates classes with a linear boundary. To fit nonlinear boundaries, you need to add polynomial terms or use other algorithms like decision trees or neural networks.

**The bias term is not about fairness.** — The bias in logistic regression means the model is unfair or prejudiced.

In statistics, bias is simply an intercept—a constant added to the raw score. It has nothing to do with fairness or discrimination. In fact: The bias (or intercept) is a learned constant that shifts the log-odds. It helps the model fit data where the baseline probability of the outcome is not 50%.


## In short

Logistic regression gives you a probability for a binary outcome. It works by combining your input numbers with learned weights and a bias to get a raw score, then squashing that score through the logistic function to get a value between 0 and 1. The weights are learned from data so that the probabilities align with actual outcomes. It is the standard starting point for any classification problem, and understanding its mechanics is a step toward more advanced models like neural networks.


## Where this is used

Logistic regression appears across many AI applications, often as a baseline or a component. In large language models (LLMs) like GPT, logistic regression is not the core architecture, but it is used in the final layer to decide, for each possible next word, the probability that it is the correct one—the same idea of turning a score into a probability. In small models that run on your phone (like keyword detectors or predictive text), logistic regression is common because it is tiny and fast, requiring only a few multiplications. In retrieval-augmented generation (RAG), when a system picks which documents to retrieve, a logistic regression might rank candidates based on relevance scores—again, outputting a probability of relevance. In agentic systems that plan and call tools, logistic regression can be used to decide whether to take an action or not, like whether to ask the user for clarification. In everyday applied machine learning, it is the go-to for churn prediction, fraud detection, medical diagnosis, credit scoring—anywhere you need a calibrated probability and interpretable coefficients. A developer often uses a library like scikit-learn, writing something like `LogisticRegression().fit(X_train, y_train)`—the library handles the math, but understanding the underlying mechanics is essential for tuning and debugging. When done badly—for example, ignoring feature scaling or using it on data with complex interactions—the model will be inaccurate. When done well, it gives you a reliable baseline and a clear story about which features matter.


---

Read this on the web: https://basavapurushottam.com/learn-ai/logistic-regression
