# Two-Class and Multiclass Classifiers

Binary classification and its extensions to many classes, via one-vs-rest, one-vs-one or natively multiclass models.

**Rests on:** Linear Regression

**Leads to:** Classification Fine-Tuning, Class Imbalance, Crop Disease Detection, K-Nearest Neighbours, Perceptron and Decision Boundaries, Rule-Based Classifiers, Sentiment Analysis, Titanic and Iris Case Studies

*Machine Learning · Machine Learning Foundations*


## Why learn this

You need to be able to sort things into groups based on what you can see or measure. Sorting into two groups, like 'spam' or 'not spam', is the basic skill. Sorting into three or more groups, like identifying a disease from a leaf image, uses that same basic skill but with more options. Understanding how these sorting tools work helps you choose the right one for your job, so you can score their accuracy correctly and predict outcomes like whether a customer will leave.


## The intuition

Imagine a piece of paper with dots on it. Each dot has a label, like 'A' or 'B'. Two-class classification is about drawing a line that puts all the 'A' dots on one side and all the 'B' dots on the other. A classifier is simply that line plus a rule: if a new dot lands on the 'A' side, call it 'A'; if it lands on the 'B' side, call it 'B'. Multiclass classification happens when there are three or more kinds of dots. One line cannot separate three groups clearly. You can use different strategies. One strategy is to draw several lines, each one separating one specific kind from all the others combined. Another strategy is to draw a line between every possible pair of kinds and let them vote on the result. A third strategy is to use a model that handles all the kinds at once without needing separate pairwise comparisons.


## An analogy

You are sorting fruit at a market. When you only have apples and oranges, you look at features like color and shape and draw a mental line: 'if it is red and round, it is an apple; if it is orange and oval, it is an orange.' This is two-class classification. Now imagine you must also sort bananas, mangoes, and grapes. You cannot use a single line for all of them. You could use the 'one-vs-rest' strategy: ask first, 'is this an apple?' If yes, set it aside. If no, ask, 'is this an orange?' Set that aside. Continue until you find what fits or declare it 'miscellaneous.' This is simple but can get confused if two fruits look similar, like a green apple and a lime. Alternatively, you could use the 'one-vs-one' strategy: compare the fruit to an apple, then to an orange, then to a banana, voting for each winner until one kind has the most votes. This is more careful but slower. Or you could develop a single mental model that looks at all features at once and directly names the fruit, which is often the best approach if you can build it.


## Definition

A classifier is a model that assigns an input to one of several categories. Two-class classification distinguishes between exactly two options, while multiclass classification distinguishes among three or more options using methods like comparing each class against all others, comparing every pair of classes, or using a single model designed for multiple groups.


## How it connects

This connects to Logistic Regression, which calculates the probability of one of two outcomes. It also relates to Decision Trees and K-Nearest Neighbours, which are tools that naturally handle multiple categories by splitting data into groups based on feature values. The concept of separating classes is similar to the dividing line between categories in Rule-Based Classifiers, but here the line is automatically learned from the data rather than manually defined.


## The mathematics

To understand how a computer decides which category an item belongs to, we need two tools: a way to measure 'agreement' between data and rules, and a way to turn that measurement into a probability.

The first tool is the dot product. Imagine you have a list of numbers describing an item (like the number of times the word 'free' appears in an email). You also have a list of numbers representing what the model has learned about that category (the weights). To find out how well the item matches the category, you multiply the first number in the data list by the first number in the weight list, then the second by the second, and so on. Finally, you add all those products together. This total sum is called the dot product. It tells you the 'raw score' or 'agreement'. If the data numbers are large where the weights are positive, the score will be high, meaning strong agreement.

For a two-class problem (like spam vs. not spam), we calculate one raw score. We take this score and add a fixed number called the bias. The bias shifts the score up or down so that the decision boundary is in the right place. Let us call this adjusted score 'z'.

A high positive z means strong evidence for class 1. A low negative z means strong evidence for class 0. But we want a probability between 0 and 1. We use a function called the sigmoid to squash the number z into that range.

The sigmoid function works like this: take the number e (approximately 2.718) raised to the power of negative z. Add 1 to that result. Then divide 1 by that sum. The formula is:

probability = 1 divided by (1 + e raised to the power of -z)

If z is a large positive number, e to the power of -z becomes very small (close to zero). The denominator becomes close to 1, so the probability is close to 1. If z is a large negative number, e to the power of -z becomes very large. The denominator becomes huge, so the probability is close to 0.

For problems with more than two classes (multiclass), we calculate a separate raw score for each class. Let us say there are C classes. For each class c, we compute a score z_c using its own weights and bias.

We cannot just use the sigmoid on each score independently because the probabilities must add up to 1. Instead, we use a function called softmax. Softmax does two things: it makes all scores positive (by raising e to the power of each score), and then it normalizes them so they sum to 1.

Here is how to calculate the probability for class c:

Step 1: Calculate the raw score z_c for class c.
Step 2: Raise e to the power of z_c. Do this for every class j from 1 to C.
Step 3: Add up all these exponential values. This is the sum of e raised to the power of z_j for all j.
Step 4: Divide the exponential value for class c by that total sum.

The formula is:

probability for class c = (e raised to the power of z_c) divided by (the sum of e raised to the power of z_j for all classes j)

This ensures that if one class has a much higher raw score than the others, its exponential will dominate the sum, resulting in a high probability. If all scores are equal, the probabilities will be equal.


## A worked example

Let us walk through the binary classification example step by step.

We have two classes: 'spam' (labeled 1) and 'not spam' (labeled 0).
The features for an email are:
x1 = 3 (the word 'free' appears 3 times)
x2 = 1 (there is 1 link)
The model has learned the following weights for the 'spam' class:
w1 = 2
w2 = -1
The bias for this class is:
b = -0.5

Step 1: Compute the dot product.
We multiply each feature by its corresponding weight and add the results.
Dot product = (w1 * x1) + (w2 * x2)
Dot product = (2 * 3) + (-1 * 1)
Dot product = 6 + (-1)
Dot product = 5
This means the features agree strongly with the 'spam' pattern.

Step 2: Add the bias to get the raw score z.
z = Dot product + b
z = 5 + (-0.5)
z = 4.5
The positive value of 4.5 indicates strong evidence for spam.

Step 3: Apply the sigmoid function to convert z into a probability.
The formula is: probability = 1 / (1 + e^(-z))
First, calculate e raised to the power of -4.5.
e^(-4.5) is approximately 0.0111
Next, add 1 to this value:
1 + 0.0111 = 1.0111
Finally, divide 1 by this sum:
1 / 1.0111 ≈ 0.989

Step 4: Make the prediction.
The probability of being spam is 0.989 (or 98.9%).
Since this probability is greater than 0.5, we predict the class is 'spam'.

Result: The email is classified as 'spam' with 98.9% confidence.


## From scratch, in Python

```python
# Learn Python building blocks first
# A list is an ordered collection of items, like [3, 1]
features = [3, 1]  # email features: 'free' count, link count
weights = [2, -1]  # learned weights for each feature
bias = -0.5

# A loop lets you repeat an operation over each item in a list.
# Here, we multiply each feature by its weight and add to a running total.
total = 0
for i in range(len(features)):
    total = total + features[i] * weights[i]

# Add bias
raw_score = total + bias

# Sigmoid function
def sigmoid(z):
    return 1 / (1 + 2.718281828459045 ** (-z))

prob = sigmoid(raw_score)
print("Probability of spam:", prob)
if prob > 0.5:
    print("Predicted class: spam")
else:
    print("Predicted class: not spam")

# For multiclass, we use a list of weight vectors.
# Let's have three classes: 'apple', 'banana', 'orange'
# Each class has its own weights and bias.
class_weights = [
    [0.5, -0.2, 0.1],  # weights for 'apple'
    [-0.3, 0.8, 0.4],  # weights for 'banana'
    [0.2, 0.1, -0.6]   # weights for 'orange'
]
class_biases = [0.1, -0.2, 0.3]

# A function to compute raw scores for all classes
def raw_scores(features, class_weights, class_biases):
    scores = []
    for c in range(len(class_weights)):
        total = 0
        for i in range(len(features)):
            total = total + features[i] * class_weights[c][i]
        scores.append(total + class_biases[c])
    return scores

# Softmax: exponentiate each score, then divide by their sum
def softmax(scores):
    exp_scores = []
    for s in scores:
        exp_scores.append(2.718281828459045 ** s)
    sum_exp = sum(exp_scores)
    probs = []
    for e in exp_scores:
        probs.append(e / sum_exp)
    return probs

# Example input: [weight, height, sweetness] for a fruit
fruit_features = [5, 3, 7]
scores = raw_scores(fruit_features, class_weights, class_biases)
probs = softmax(scores)
print("Class probabilities:", probs)
# Largest probability is the predicted class
predicted_class = probs.index(max(probs))
print("Predicted class index:", predicted_class)
```


## What the code does

The plain Python code starts by defining a list of features and weights. A list is a collection of items in a specific order, so [3, 1] means the first feature is 3 and the second is 1. The loop iterates over each index i from 0 to 1 (the length of features), multiplies features[i] by weights[i], and adds to a total. This is the dot product. Then we add the bias. The sigmoid function takes any number and returns a value between 0 and 1, representing probability. The if-else statement decides the class based on a threshold of 0.5. For the multiclass part, we define a list of weight lists (a 2D array) where each inner list corresponds to a class. The raw_scores function computes a score for each class by doing a dot product plus bias, looping through each class and each feature. The softmax function exponentiates each score, then divides by the sum of all exponentiated scores, giving a set of probabilities that add up to 1. The index of the largest probability is the predicted class. In the NumPy version, np.dot does the same multiplication and addition loop for you, and np.exp applies the exponential to every element. Scikit-learn's LogisticRegression wraps all this: you just fit and predict, and it decides between one-vs-rest and multinomial (native multiclass) based on the solver and data.


## Common misconceptions

**Accuracy is always the right metric** — If my model gets 95% accuracy, it's doing well.

If 95% of emails are not spam, a model that always says 'not spam' gets 95% accuracy without learning anything. Accuracy hides class imbalance. In fact: For imbalanced data, use precision and recall, or the F1 score. Accuracy is only meaningful when classes are roughly balanced.

**One-vs-rest scores are probabilities you can compare** — In one-vs-rest, the score for 'cat' is 0.8 and for 'dog' is 0.6, so the model is more confident about 'cat'.

Each one-vs-rest classifier is trained independently, so the scores are not calibrated against each other. 0.8 for cat and 0.6 for dog don't mean the same thing. In fact: Use softmax (which forces all probabilities to sum to 1) or calibrate the scores before comparing across classes.

**Multiclass is just many binary classifiers** — To do multiclass, just train a binary classifier for each class and pick the highest score.

That's one approach (one-vs-rest), but it ignores correlations between classes and can be slow. Natively multiclass models (like softmax regression or neural networks) often do better. In fact: One-vs-rest is a valid baseline, but native multiclass models learn a single decision boundary and share information across classes, which can improve accuracy.

**Binary classification requires exactly two features** — To classify, you need to draw a line, which requires two numbers (x and y) on a graph.

Features can be many more than two. The decision boundary is a hyperplane in high-dimensional space, not a line on paper. In fact: A classifier works with any number of features — hundreds or even millions. The math scales up, but the idea is the same.

**Sigmoid output is the same as a probability** — The sigmoid output is a probability, so I can directly compare it to another model's sigmoid output.

Sigmoid outputs are only calibrated if the model is well-calibrated. Many models aren't, so 0.8 from one model may not mean the same as 0.8 from another. In fact: Treat sigmoid outputs as scores, not calibrated probabilities. Calibrate them (e.g., with Platt scaling) if you need true probabilities.


## In short

Two-class classification assigns an input to one of two categories; multiclass extends this to many categories. The three main strategies: one-vs-rest (train one binary classifier per class), one-vs-one (train a binary classifier per pair and vote), and native multiclass (a single model predicts all classes, typically with a softmax layer). The mathematics: dot product plus bias gives a raw score; sigmoid squashes to a probability for binary; softmax normalizes scores to a probability distribution for multiclass. In code, this is a few loops and a softmax function, but libraries like scikit-learn handle it for you. Watch out for class imbalance and uncalibrated scores; always choose metrics that match your problem.


## Where this is used

In large language models (LLMs) like GPT-4, classification is everywhere. The model predicts the next token (word or subword) by computing a probability distribution over the entire vocabulary — that's a multiclass classification with tens of thousands of classes. The final layer is a softmax over the vocabulary, and the output is the most probable token (or a sample from the distribution). In small language models running on phones or laptops (e.g., on-device next-word prediction), the same architecture is used but with a smaller vocabulary and fewer parameters, so the softmax is cheaper. One-vs-rest is less common because the vocabulary is too large to train thousands of independent binary classifiers, but some efficient approximations use hierarchical softmax, which is like a tree-structured one-vs-rest. In retrieval-augmented generation (RAG), a retriever selects relevant documents from a large corpus. This is often a binary or multiclass classifier: for each document, it predicts whether it is relevant to the query. The retriever scores documents and picks the top-k; a binary relevance classifier is trained on positive (relevant) and negative (irrelevant) pairs. In agentic systems that plan and call tools, the LLM must decide which tool to use (e.g., search, calculator, code interpreter) — that's a multiclass classification over available tools. The LLM's output is a probability distribution over tool names, and the system picks the highest or samples. In everyday applied machine learning, two-class and multiclass classifiers are everywhere: spam detection (spam vs not), fraud detection (fraud vs normal), medical diagnosis (disease vs healthy), image recognition (classify objects into many categories), and churn prediction (will a customer leave or stay). A developer using scikit-learn writes: `from sklearn.ensemble import RandomForestClassifier; model = RandomForestClassifier(); model.fit(X_train, y_train); preds = model.predict(X_test)`. RandomForest handles multiclass natively by voting among trees. When things go wrong, it's often due to class imbalance: if one class is rare (e.g., 1% fraud), the model may just predict the majority class and get 99% accuracy, so you need metrics like precision/recall and techniques like class weights or oversampling. One-vs-rest can produce uncalibrated scores — the probabilities from different binary classifiers aren't comparable, so you can't trust the confidence. Softmax gives proper probabilities, but if the model is miscalibrated, even those need calibration (e.g., Platt scaling). Developing software with these models involves choosing the right metric: accuracy is misleading for imbalanced data; precision and recall are often better. It also involves understanding the trade-offs between one-vs-rest (simple, but slower for many classes) and one-vs-one (more accurate but more expensive) and native multiclass (best but harder to interpret).


---

Read this on the web: https://basavapurushottam.com/learn-ai/two-class-and-multiclass-classifiers
