# Gini and Information Gain

The impurity criteria trees use to choose splits — Gini impurity and entropy-based information gain.

**Rests on:** Decision Trees, Mutual Information

*Machine Learning · Machine Learning Foundations*


## Why learn this

This unlocks **Decision Trees**, the first model in your library that you can read like a rulebook. From there you'll move to **Random Forests** and **Gradient Boosting** — the algorithms that win most machine-learning competitions on tabular data. You'll also need this before **Churn Prediction** and **Crop Disease Detection**, since both rely on trees. Understanding how a tree decides where to split is the difference between trusting a model and understanding it.


## The intuition

Imagine you're sorting a box of mixed fruit — apples and oranges — into two trays. You can't see inside the box, but you can feel each fruit. Your goal: after one feel, put as many of the same fruit on each tray as possible. Before you feel anything, the box is a messy mix. After you sort, each tray is cleaner — mostly apples here, mostly oranges there. The 'messiness' of a tray is what we call **impurity**. Gini impurity and information gain are two different tape measures for that mess. A good split is one that makes each tray much less messy than the original box. Gini asks: 'If I pick a fruit at random from this tray and then guess its kind by the majority in the tray, how often will I be wrong?' Entropy (used in information gain) asks: 'How much surprise is there in this tray?' Both give you a number: 0 means perfectly clean, higher numbers mean messier. The tree tries every possible split and picks the one that gives the biggest drop in mess — that drop is the 'information gain'.


## An analogy

Think of a teacher sorting students into two study groups based on one yes/no question, trying to separate those who will pass an exam from those who won't. The teacher can ask: 'Did you sleep more than 6 hours?' or 'Do you have a study plan?' or 'Do you own a pet?' The teacher doesn't know the answers yet, but she has last year's data. She computes: for each question, if I split the students by the answer, how pure would each group be? A question that perfectly separates pass/fail gives pure groups — no mess. A question that leaves both groups half pass, half fail gives no gain — the split is useless. The teacher picks the question with the biggest gain. This works because impurity is measurable: it's the opposite of 'purity', a number from 0 (all same) to some maximum (perfectly mixed). Where does this analogy break? In reality, a split rarely gives pure groups, and the tree often needs many splits to reach useful purity. Also, the tree doesn't ask a question once — it recurses, asking new questions on each subgroup, building a tree. And the tree doesn't choose questions that are 'meaningful' to humans; it simply optimises the arithmetic of impurity. So the teacher analogy is a clean story, but the real tree is a relentless number-cruncher that will split on any feature if it reduces mess, even if the question seems silly to us.


## Definition

Gini impurity and entropy-based information gain are two mathematical measures of how mixed (impure) the classes are within a node of a decision tree, used to decide which feature and threshold to split on at each step.


## How it connects

This is part of **Machine Learning Foundations**, under the umbrella of learning functions from data. You've already met **Decision Trees** as a model family (interpretable, but prone to overfitting). This concept gives you the precise arithmetic inside a tree's split decision. It also connects to **K-Nearest Neighbours** — both are non-parametric, meaning they don't learn fixed coefficients, but trees use impurity to partition space, not distance. And this leads to **Decision Jungles**, a compressed variant that also uses impurity measures to choose merges.


## The mathematics

This concept rests on **probability** and **arithmetic**. Probability comes in because we're measuring the proportion of each class in a group. Arithmetic because all we do is multiply, add, and take a logarithm (a fancy word for the inverse of exponentiation — you'll see it in the formula stage). Let's build the idea in words first. **Gini impurity**: If you pick one item at random from a group, then guess its class by randomly choosing from the classes present (with probability equal to their frequency), Gini impurity is the probability you get it wrong. The formula: for each class, take its proportion, multiply by (1 minus its proportion), and sum all those products. If all items are the same class, one proportion is 1, others are 0, so you get 0 — pure. If there are two classes at 50-50, you get 0.5 — maximum impurity. **Entropy**: This comes from information theory — how much 'surprise' or 'uncertainty' is in the group. If a group is all one class, there's no surprise (0 entropy). If it's half and half, there's maximum surprise (1 bit for two classes). The formula: for each class, multiply its proportion by the logarithm base 2 of that proportion, take the negative, and sum. Then **information gain** is simply: the impurity of the parent node minus the weighted average of the impurities of the child nodes after a split. Higher gain means the split created cleaner children. We'll now do this with numbers.

Gini impurity: Gini(p) = Σ p_i * (1 - p_i) for each class i. Entropy: E(p) = -Σ p_i * log2(p_i). Information gain: Gain = I_parent - Σ (N_child / N_parent) * I_child, where I is either Gini or Entropy.


## A worked example

```
Suppose we have a node with 10 customers: 6 churn (C) and 4 stay (S). We're considering a split by 'has contract' (yes/no). After split: Left child (yes): 2 C, 4 S. Right child (no): 4 C, 0 S.

Step 1: Parent impurity (Gini): (6/10)*(1-6/10) + (4/10)*(1-4/10) = 0.6*0.4 + 0.4*0.6 = 0.24 + 0.24 = 0.48   # Compute Gini impurity of the parent node using the formula: sum of p*(1-p) over classes.
Step 2: Left child Gini: (2/6)*(1-2/6) + (4/6)*(1-4/6) = 0.333*0.667 + 0.667*0.333 = 0.222 + 0.222 = 0.444   # Compute Gini for the left child (yes group): 6 items, 2 churn, 4 stay.
Step 3: Right child Gini: (4/4)*(1-4/4) + (0/4)*(1-0/4) = 1*0 + 0*1 = 0   # Right child is pure (all churn), so Gini = 0.
Step 4: Weighted average of child impurities: (6/10)*0.444 + (4/10)*0 = 0.2664 + 0 = 0.2664   # Each child's impurity is weighted by its fraction of the total.
Step 5: Information gain (Gini reduction): 0.48 - 0.2664 = 0.2136   # Gain = parent impurity minus weighted child impurities.

Result: The split on 'has contract' reduces Gini impurity from 0.48 to 0.2664, giving a gain of 0.2136. Higher gain = better split. Compare splits and pick the one with the highest gain.
```


## From scratch, in Python

```python
# Gini impurity and information gain from scratch

def gini(labels):
    """labels: list of class labels (strings or ints). Returns Gini impurity."""
    total = len(labels)
    if total == 0:
        return 0.0
    # Count how many of each label
    counts = {}
    for label in labels:
        counts[label] = counts.get(label, 0) + 1
    impurity = 0.0
    for label in counts:
        p = counts[label] / total
        impurity += p * (1 - p)
    return impurity

def entropy(labels):
    """labels: list of class labels. Returns entropy in bits."""
    import math
    total = len(labels)
    if total == 0:
        return 0.0
    counts = {}
    for label in labels:
        counts[label] = counts.get(label, 0) + 1
    ent = 0.0
    for label in counts:
        p = counts[label] / total
        ent -= p * math.log2(p)
    return ent

def info_gain(parent_labels, children_labels, criterion='gini'):
    """children_labels: list of lists. Each list contains labels for a child node."""
    if criterion == 'gini':
        impurity = gini
    elif criterion == 'entropy':
        impurity = entropy
    else:
        raise ValueError("criterion must be 'gini' or 'entropy'")
    
    parent_gain = impurity(parent_labels)
    n_parent = len(parent_labels)
    weighted_children = 0.0
    for child in children_labels:
        weighted_children += (len(child) / n_parent) * impurity(child)
    return parent_gain - weighted_children

# Example from the maths section
parent = ['C', 'C', 'C', 'C', 'C', 'C', 'S', 'S', 'S', 'S']  # 6 churn, 4 stay
left_child = ['C', 'C', 'S', 'S', 'S', 'S']  # 2 C, 4 S
right_child = ['C', 'C', 'C', 'C']  # 4 C, 0 S

print("Gini of parent:", gini(parent))  # should be 0.48
print("Gini of left:", gini(left_child))  # 0.444
print("Gini of right:", gini(right_child))  # 0.0
print("Gain (gini):", info_gain(parent, [left_child, right_child], 'gini'))  # 0.2136
print("Entropy of parent:", entropy(parent))  # ~0.971
print("Gain (entropy):", info_gain(parent, [left_child, right_child], 'entropy'))
```


## The same thing, with a framework

```python
# In scikit-learn, you don't write these functions yourself.
# The DecisionTreeClassifier does it internally, but you can inspect the chosen splits.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100, n_features=4, random_state=42)
clf = DecisionTreeClassifier(criterion='gini', max_depth=2)
clf.fit(X, y)

# See which feature was used for the first split and at what threshold
print("Split feature:", clf.tree_.feature[0])
print("Split threshold:", clf.tree_.threshold[0])

# To see the impurity reduction at each node:
print("Impurity at root:", clf.tree_.impurity[0])
print("Impurity at left child:", clf.tree_.impurity[1])

# The criterion parameter can be 'gini' or 'entropy' — try both and compare accuracy.
```


## What the code does

**From scratch code** — Line by line: First we define `gini(labels)`. It takes a list of labels (like `['C', 'C', 'S']`). We count how many of each label using a dictionary. A dictionary is a collection that maps keys to values — here label names to counts. Then for each label we compute the proportion p and add p*(1-p) to the impurity. This mirrors the maths exactly. Then `entropy` does the same, but uses `math.log2(p)` (log base 2) and subtracts. Then `info_gain` takes the parent labels and a list of child label lists. It computes the parent impurity, then the weighted sum of child impurities, and subtracts. The example at the bottom creates the same numbers as in the maths section and prints them, so you can verify 0.48 and 0.2136. **Framework code** — In scikit-learn, you just set `criterion='gini'` or `'entropy'` when creating the tree. Under the hood, scikit-learn computes the same impurity reductions for every possible threshold across every feature, using optimised Cython code. But you can read off what it did: `tree_.feature[0]` tells you which feature was split on, `tree_.threshold[0]` the value, and `tree_.impurity` shows the impurities after each split. The arguments `max_depth` and others control overfitting — a deep tree will keep splitting until pure, which often overfits.


## Common misconceptions

**Gini and entropy always give the same tree** — Since they rarely disagree materially, I can just use whichever and never worry about the difference.

They measure impurity differently and can produce different split choices in borderline cases. Gini is fast to compute and is the default, but entropy can lead to slightly more balanced trees in some cases. The difference is usually small, but it is not zero; it matters when you are searching for the best split in a large feature space. In fact: They usually agree, but not always — test both on your data and choose by cross-validated performance.

**Higher information gain always means better model** — The split with the highest gain is the one I should always make.

A single split's gain is not the only factor. Unrestrained splitting leads to overfitting. Also, information gain is biased toward features with many possible values — a high-cardinality feature can look like it gives great gain just by memorising, not generalising. In fact: Information gain helps choose a split, but you must control tree depth and watch for feature cardinality bias. Use techniques like gain ratio or limit depth.

**Impurity measures are only for classification, not regression** — Since trees can also predict numbers, I can use Gini for regression too.

Gini and entropy are defined for class labels. For regression, there are different criteria like variance reduction (mean squared error). Using Gini on continuous targets is meaningless because there are no class proportions. In fact: For regression trees, use variance-based criteria, not Gini or entropy.

**Entropy is the same as probability** — Entropy just means probability of an event.

Entropy is a summary of the entire probability distribution, not a single probability. It tells you how spread out or concentrated the distribution is. A single probability doesn't tell you that. In fact: Entropy is a measure of uncertainty across all possible outcomes of a distribution.

**A pure node is always good** — If I just keep splitting until every leaf is pure, I get the perfect model.

Pure leaves on training data often mean overfitting — the tree has memorised noise and won't generalise to new data. Also, pure leaves can be a sign of data leakage or too-high feature cardinality. In fact: Impurity reduction should be weighed against generalisation; limit depth and prune.


## In short

At every split, a decision tree picks the feature and threshold that most reduces 'messiness' (impurity). Gini impurity and entropy both measure how mixed the classes are; information gain is the drop in impurity after a split. Gini is faster, entropy is more information-theoretic, and they usually agree. This is the heart of tree-based models — the most widely used classical ML algorithms for tabular data. Remember: depth control beats endless purity.


## Where this is used

This concept is not just theory — it's the engine inside many real systems. **In large language models (LLMs)**: Individually, they are not built on trees, but tree-based models are often used in the *pipeline* around them — for instance, to classify prompt intent or to filter unsafe content before hitting the LLM, because trees are fast and interpretable. The mechanism: a tree uses Gini to split on tokens or other features to decide whether a prompt is 'safe' or 'toxic'. **In small models running on devices**: TinyML models for wake-word detection (like 'Hey Siri') often use decision trees or ensembles. Impurity measures are cheap to compute, so the tree can be trained offline with Gini, then deployed as a simple if-else rule that runs even on a microcontroller. The split thresholds become hard-coded comparisons. **In retrieval-augmented generation (RAG)**: Before an LLM answers, it must retrieve relevant documents. Re-ranking retrieved chunks can be done with a gradient-boosted tree (XGBoost) that scores chunk relevance. The tree's splits use information gain to separate 'relevant' from 'irrelevant' chunks based on features like overlap with the query. **In agentic systems that plan and call tools**: When an agent decides which tool to call next, it often uses a classifier to rank options. Trees are used for fast, low-power decisioning, especially in deterministic sub-routines. The mechanism: each tool call is a leaf, and the path is a series of impurity-based splits on context features. **In everyday applied machine learning**: This is where trees shine. **Churn prediction** — banks and telecoms use decision trees to identify customers likely to leave. The tree splits on features like 'months since last login' or 'number of support calls'. **Fraud detection** — credit card companies use random forests; each split reduces impurity to isolate fraud patterns. **Medical diagnosis** — trees help flag disease from symptoms, with each split a test result. **Recommendation systems** — some use trees to segment users. **What a developer actually writes**: In Python, they write `from sklearn.tree import DecisionTreeClassifier` then `clf = DecisionTreeClassifier(criterion='gini', max_depth=5).fit(X, y)`. For bigger data, they use `xgboost` or `lightgbm`, which use improvements on information gain but the core is the same. **What breaks when done badly**: If you let the tree split until pure, it overfits — it memorises noise. If you use a feature with many categories (like customer ID), information gain can be biased toward it, because splitting on it creates many tiny pure nodes — this is the known 'gain ratio' problem. If you ignore missing values or data leakage, the split choices are misleading. So practitioners must control depth, handle categoricals carefully, and validate.


---

Read this on the web: https://basavapurushottam.com/learn-ai/gini-and-information-gain
