# Random Forests

Ensembles of decorrelated decision trees, each trained on a bootstrap sample with a random feature subset, averaged for prediction.

**Rests on:** Decision Trees

*Machine Learning · Machine Learning Foundations*


## Why learn this

Random forests are one of the most reliable tools for machine learning. They provide a strong starting point for almost any dataset organized in rows and columns, where each row is an example and each column is a piece of information about that example. This structure is called a tabular problem because the data looks like a spreadsheet. Random forests help you predict outcomes for new examples and tell you which pieces of information (features) are most important. They rarely require delicate tuning, meaning you can often get good results without spending hours adjusting settings. Learning this builds a solid foundation for understanding more complex systems. For instance, the idea of combining many simple predictions into one reliable result is a key ingredient in modern techniques that power advanced language models, which generate text based on patterns in massive amounts of writing.


## The intuition

A single decision tree can be unstable. A decision tree is a model that makes predictions by asking a series of yes-or-no questions about the data, following a path from the top to a final answer. If you change the training data even slightly, the structure of this tree can change completely, causing its predictions to jump around unpredictably. This instability happens because a single tree tries to memorize every detail of the specific data it sees.

Random forests fix this problem by building many decision trees and combining their answers. Instead of relying on one tree, the forest creates a large number of trees. Each tree is trained on a slightly different version of the data, created by picking examples randomly with replacement (meaning an example can be picked more than once, while others might not be picked at all). Additionally, when each tree decides which question to ask next, it only looks at a random subset of the available features. This double randomness ensures that the trees are different from one another.

For example, imagine you want to predict whether a customer will buy a product. One tree might focus heavily on age, another on past purchase history, and another on time spent on the site, because they saw different data or looked at different features. When you combine these diverse opinions, the errors of individual trees tend to cancel each other out, resulting in a more reliable and often more accurate prediction than any single tree could provide.


## An analogy

Think of a committee of experts making a decision, such as a group of doctors diagnosing a patient’s condition. If you asked just one doctor for a diagnosis, their answer might depend heavily on their specific training, the recent medical journals they read, or even the time of day. Their judgment could be influenced by rare cases they recently treated, leading to inconsistency.

However, if you assemble a diverse panel of doctors and ask each one to diagnose the patient independently, the group decision is usually more balanced and robust. This is how a random forest works. Each tree in the forest is like one doctor. The 'bootstrap sample' means that each doctor reviews a different set of past patient records (some records appear multiple times for one doctor but not at all for another). The 'random feature subset' means that when each doctor thinks about the diagnosis, they are only allowed to look at a random selection of the patient’s symptoms and test results, ignoring others temporarily. The final diagnosis is determined by the majority vote of all doctors.

Where this analogy has limits: If all the doctors were trained identically and looked at the exact same data, they would all make the same mistakes in the same way. Random forests avoid this by shuffling the data and limiting the information available to each tree, ensuring diversity. Also, while a single doctor can clearly explain their reasoning step-by-step, a committee’s collective decision is harder to trace back to a single clear cause, so random forests trade some interpretability for accuracy.


## Definition

A random forest is a machine learning model that combines the predictions of many decision trees. Each tree is trained on a random sample of the data and uses only a random subset of features at each step. For problems where you predict a number (regression), the final result is the average of all trees' outputs; for problems where you predict a category (classification), the final result is the most common choice among all trees.


## How it connects

You have notes on Decision Trees, which are the individual building blocks here. A random forest is essentially many decision trees working together, but with crucial tweaks: each tree gets a different random sample of the data and considers only a random subset of features at each split. This diversity allows the average prediction to be more stable than any single tree. You can compare this to K-Nearest Neighbours, another method that predicts based on similar past examples, but while K-Nearest Neighbours looks at local similarities for every new question, a random forest combines many rigid, local rules into a broader consensus. This approach is also your first exposure to ensemble methods, which are techniques that combine multiple simple models to create a more powerful one.


## The mathematics

This concept relies on basic arithmetic: counting, adding numbers, and finding averages. You do not need calculus or advanced algebra.

First, we must understand how the data is prepared for each tree in the forest. We use a method called bootstrap sampling. This means we take your original list of data rows and draw a new list of the same length from it, but we allow the same row to be picked more than once. If a row is picked, it stays in the pool, so it can be picked again. Some rows will appear multiple times in this new list; others may not appear at all.

Second, when we build each tree, we do not look at every feature (column of data) available. At each step where the tree needs to decide how to split the data, we randomly choose a small number of features to consider. We call this number m. A common rule is to set m to be the square root of the total number of features p. For example, if you have 100 features, m might be 10. This forces each tree to look at different aspects of the data, making the trees different from one another.

Third, we combine the predictions of all the trees. We grow B trees. Let us call them Tree 1, Tree 2, ..., up to Tree B. When a new case comes in, every tree gives its own answer. 

For regression (predicting a number), we add up all B answers and divide by B. This is the average.

For classification (predicting a category), we count how many trees voted for each category. The category that gets the most votes wins. This is called a majority vote.

Why does this work? A single tree can be very sensitive to the specific data it sees. It might memorize noise in the training data, which hurts its performance on new data. This is called overfitting. By averaging many trees that are different from each other, the random errors (noise) tend to cancel out. The true signal, which is consistent across trees, remains. Therefore, the average prediction is usually more accurate than any single tree's prediction.

The formula for the final prediction depends on the task:

For regression, if we have B trees, and Tree b gives a prediction f_b(x) for input x, the forest prediction y_hat is:
y_hat = (f_1(x) + f_2(x) + ... + f_B(x)) divided by B.

For classification, let C_b(x) be the class predicted by Tree b. The forest prediction is the class that appears most frequently in the list {C_1(x), C_2(x), ..., C_B(x)}.


## A worked example

Let us walk through a tiny example with real numbers.

Setup:
We have 5 loan applications. Each has two features: Income (in thousands) and Credit Score (0-100). The target is Repaid (1) or Defaulted (0).
Original Data Rows:
Row 1: Income=50, Credit=70, Repaid=1
Row 2: Income=60, Credit=80, Repaid=1
Row 3: Income=40, Credit=60, Repaid=0
Row 4: Income=80, Credit=90, Repaid=1
Row 5: Income=30, Credit=50, Repaid=0

We will build a Random Forest with B = 3 trees.

Step 1: Create Bootstrap Sample for Tree 1.
We draw 5 rows from the original 5, with replacement. Let us say the random draws give us these row indices: [2, 1, 2, 4, 2].
This means Tree 1 sees:
- Row 2 (Income=60, Credit=80, Repaid=1)
- Row 1 (Income=50, Credit=70, Repaid=1)
- Row 2 again (Income=60, Credit=80, Repaid=1)
- Row 4 (Income=80, Credit=90, Repaid=1)
- Row 2 again (Income=60, Credit=80, Repaid=1)
Notice that Row 3 and Row 5 are missing. Row 2 appears three times.

Step 2: Create Bootstrap Sample for Tree 2.
Let us say the draws give indices: [3, 1, 5, 1, 3].
Tree 2 sees:
- Row 3 (Income=40, Credit=60, Repaid=0)
- Row 1 (Income=50, Credit=70, Repaid=1)
- Row 5 (Income=30, Credit=50, Repaid=0)
- Row 1 again (Income=50, Credit=70, Repaid=1)
- Row 3 again (Income=40, Credit=60, Repaid=0)

Step 3: Create Bootstrap Sample for Tree 3.
Let us say the draws give indices: [4, 4, 2, 1, 5].
Tree 3 sees:
- Row 4 (Income=80, Credit=90, Repaid=1)
- Row 4 again (Income=80, Credit=90, Repaid=1)
- Row 2 (Income=60, Credit=80, Repaid=1)
- Row 1 (Income=50, Credit=70, Repaid=1)
- Row 5 (Income=30, Credit=50, Repaid=0)

Step 4: Train the Trees.
We assume each tree is trained on its specific bootstrap sample. We also apply the feature subsampling rule. Suppose there are p=2 features (Income, Credit). We set m to be the square root of 2, which is approximately 1.41. We round down or choose 1 feature randomly at each split. For simplicity, let us assume Tree 1 uses only Income for its main split, Tree 2 uses only Credit, and Tree 3 uses only Income.

After training (which involves complex splitting logic not shown here), suppose the trees produce these simple decision rules:
- Tree 1 predicts Repaid (1) if Income > 55. Otherwise Defaulted (0).
- Tree 2 predicts Repaid (1) if Credit > 75. Otherwise Defaulted (0).
- Tree 3 predicts Repaid (1) if Income > 65. Otherwise Defaulted (0).

Step 5: Predict for a New Applicant.
New Applicant: Income=55, Credit=75.

Tree 1 Prediction:
Check Income: Is 55 > 55? No.
Prediction f_1 = 0 (Defaulted).

Tree 2 Prediction:
Check Credit: Is 75 > 75? No.
Prediction f_2 = 0 (Defaulted).

Tree 3 Prediction:
Check Income: Is 55 > 65? No.
Prediction f_3 = 0 (Defaulted).

Step 6: Combine Predictions.
For classification, we take a majority vote.
The votes are: [0, 0, 0].
Count of 0s: 3.
Count of 1s: 0.
The majority is 0.

Result:
The Random Forest predicts the loan will be Defaulted (0).

Note: In this specific case, all trees agreed. In other cases, they might disagree. If the votes were [1, 1, 0], the majority would be 1 (Repaid). The forest always takes the most common answer among its trees.


## From scratch, in Python

```python
import random

# Build a simple decision tree
class TreeNode:
    def __init__(self, feature=None, threshold=None, left=None, right=None, value=None):
        self.feature = feature
        self.threshold = threshold
        self.left = left
        self.right = right
        self.value = value  # For leaf: the majority class

def fit_tree(X, y, features_available):
    # If all same class or no features left, return leaf
    if len(set(y)) == 1 or not features_available:
        return TreeNode(value=max(set(y), key=y.count))
    # Simple heuristic: pick the best split among randomly selected features
    best_gini = float('inf')
    best_criteria = None
    for feature in features_available:
        thresholds = set(x[feature] for x in X)
        for threshold in thresholds:
            left_mask = [i for i, x in enumerate(X) if x[feature] < threshold]
            right_mask = [i for i in range(len(X)) if i not in left_mask]
            if len(left_mask) == 0 or len(right_mask) == 0:
                continue
            # Compute Gini impurity (simplified)
            gini = 0
            for mask in (left_mask, right_mask):
                classes = [y[i] for i in mask]
                if not classes:
                    continue
                p = classes.count(1)/len(classes)
                gini += (p*(1-p)*len(classes)/len(X))
            if gini < best_gini:
                best_gini = gini
                best_criteria = (feature, threshold)
    if best_criteria is None:
        return TreeNode(value=max(set(y), key=y.count))
    feature, threshold = best_criteria
    left_mask = [i for i, x in enumerate(X) if x[feature] < threshold]
    right_mask = [i for i in range(len(X)) if i not in left_mask]
    left = fit_tree([X[i] for i in left_mask], [y[i] for i in left_mask], features_available)
    right = fit_tree([X[i] for i in right_mask], [y[i] for i in right_mask], features_available)
    return TreeNode(feature, threshold, left, right)

def predict_tree(node, x):
    if node.value is not None:
        return node.value
    if x[node.feature] < node.threshold:
        return predict_tree(node.left, x)
    else:
        return predict_tree(node.right, x)

def bootstrap_sample(X, y):
    n = len(y)
    indices = [random.randint(0, n-1) for _ in range(n)]
    return [X[i] for i in indices], [y[i] for i in indices]

def random_forest_fit(X, y, n_trees=10, max_features=None):
    n_features = len(X[0])
    if max_features is None:
        max_features = int(n_features**0.5)  # sqrt for classification
    forest = []
    for _ in range(n_trees):
        X_sample, y_sample = bootstrap_sample(X, y)
        # Randomly select features for this tree
        available_features = random.sample(range(n_features), max_features)
        tree = fit_tree(X_sample, y_sample, available_features)
        forest.append(tree)
    return forest

def random_forest_predict(forest, x):
    # Majority vote
    votes = [predict_tree(tree, x) for tree in forest]
    return max(set(votes), key=votes.count)

# Example usage
# X is a list of lists, y is a list of labels
# X = [[50,70],[60,80],[40,60],[80,90],[30,50]]
# y = [1,1,0,1,0]
# Then run forest = random_forest_fit(X, y, n_trees=3, max_features=1)
# print(random_forest_predict(forest, [55, 75]))
```


## The same thing, with a framework

```python
from sklearn.ensemble import RandomForestClassifier

# Create the model
rf = RandomForestClassifier(n_estimators=100, max_features='sqrt', random_state=42)
# Train it on features X (list of lists) and labels y (list)
rf.fit(X, y)
# Predict for a new sample
prediction = rf.predict([[55, 75]])
```
This is equivalent to the from-scratch code: `n_estimators` is the number of trees, `max_features` controls the feature subset size, and `fit` builds the forest.


## What the code does

We'll explain the from-scratch code in detail. First, `TreeNode` is a simple structure that stores a feature, a threshold, left/right children, and a value. The `fit_tree` function builds a decision tree recursively. It takes the data, the labels, and a list of available features (the random subset). It checks if all labels are the same, then it creates a leaf. Otherwise, it tries every feature and threshold in the available set, computing a simple Gini impurity to pick the best split. Then it splits the data and recursively builds left and right children. `predict_tree` walks down the tree to find a leaf and returns its value. `bootstrap_sample` creates a new dataset by sampling rows with replacement. `random_forest_fit` creates several trees, each with a different bootstrap sample and a random feature subset. `random_forest_predict` averages the trees' predictions (here majority vote). The framework code does all this in three lines, but you should understand the underlying mechanisms: many trees, bootstrapping, and feature randomness.


## Common misconceptions

**More trees always make it better** — If 100 trees are good, 1000 are better, and 10000 even better.

Accuracy improves quickly until a point, then plateaus. After hundreds of trees, the gain is negligible, but training and prediction time keep growing. Also, too many trees can hurt if each tree is overfitting, though averaging usually helps. In fact: A random forest is a smoothing operation; using enough trees to get a stable prediction is key, but beyond that the returns diminish. In practice, 100-500 trees is common.

**Random forests don't overfit** — Averaging many trees means the model can't memorize the data, so overfitting is impossible.

Averaging reduces variance but does not eliminate bias. If each tree is extremely deep and the data is noisy, the forest can still fit the noise. But because of the bootstrapping and feature randomness, it overfits less than a single unpruned tree. In fact: Random forests are more robust than single trees but can still overfit if the trees are too deep or the data is very noisy. Regularization like limiting depth or setting minimum samples per leaf helps.

**All trees see the same data** — Each tree trains on the full dataset, just with different starting points.

If all trees saw the same data, they would be highly correlated, and averaging would not help much. The magic of random forests comes from diversity: bootstrap samples and random feature subsets make each tree different. In fact: Each tree trains on a bootstrap sample (a random subset with replacement), and at each split considers only a random subset of features. This decorrelates the trees, making the ensemble effective.

**Feature importance is always trustworthy** — The feature importance scores from a random forest tell me exactly which features cause the target.

Feature importance measures how often a feature is used in splits and how much it reduces impurity, but it can be biased towards features with many categories or correlated features. It is a useful heuristic, not a causal statement. In fact: Use feature importance as an initial screening tool, but confirm with domain knowledge or permutation importance for a more reliable measure.


## In short

Random forests are ensembles of decision trees, each trained on a bootstrap sample and using random feature subsets. Averaging their predictions reduces variance and improves accuracy compared to a single tree. They are a powerful, versatile tool for tabular data, ranking among the best classical models. The key to their success is diversity among trees, achieved by randomness in both rows and features. Training is fast, and prediction is straightforward. They provide feature importance and out-of-bag error for free. Remember: they are not a black box if you use them with care, but they are not immune to overfitting. This is a foundation for understanding ensemble methods and boosting.


## Where this is used

Random forests are used across many applied machine learning domains. In **everyday tabular problems**, such as churn prediction, credit scoring, or medical diagnosis, they are often a strong first choice. For example, your notes on Churn Prediction could use a random forest to estimate which customers will leave, because it handles mixed features and missing data well. The developer would use `RandomForestClassifier` from scikit-learn, fine-tuning `n_estimators` (number of trees) and `max_depth`. In **retrieval-augmented generation**, random forests themselves aren't directly used, but the same idea of ensemble and feature randomness appears in ranking models that combine many signals. Also, random forests can be used to create embeddings of data points (e.g., leaf-vector embeddings) that feed into other models. In **small models on devices**, like your notes on Decision Jungles, random forests are sometimes converted to decision jungles or used directly if the tree size is manageable, providing a lightweight model for keyword spotting or activity recognition. A developer would train a random forest on a server and then export the tree structure to be executed on a phone. In **agentic systems**, random forests are less central; agents typically use larger models, but they might use a random forest as a tool to make predictions on structured data within a loop. For example, an agent that predicts inventory demand, or a fraud detection agent, could query a random forest that runs locally, fast and without external calls. In **AI safety**, random forests are used for uncertainty estimation — the variance across trees gives a sense of confidence, which can trigger fallback or human intervention. **What developers actually write**: they load a CSV, split it into train/test, fit a random forest, and evaluate metrics like accuracy and AUC. The biggest pitfall is not tuning `max_features` or `min_samples_leaf`. Done badly, a random forest can overfit noisy data or take too long to predict on a mobile device. Done well, it is a reliable, interpretable (via feature importance) model that works out of the box.


---

Read this on the web: https://basavapurushottam.com/learn-ai/random-forests
