# Decision Jungles

Decision forests whose trees are directed acyclic graphs rather than trees, letting paths merge and reducing memory.

**Rests on:** Decision Trees, Topological Sort

*Machine Learning · Machine Learning Foundations*


## Why learn this

You want to build smart systems that work on a phone, a watch, or a sensor without needing an internet connection. To do this, you must shrink your models so they fit into limited memory and run quickly. A decision jungle lets you achieve this by removing redundant parts of the model, keeping it small and fast while maintaining its ability to make correct decisions. This is the practical difference between a concept that needs a server room and a product that lives in your pocket.


## The intuition

Think of how a standard decision tree works. It is like a flowchart where every possible route starts at the top and goes down, never crossing paths. If two different situations lead to the same conclusion, the tree builds two entirely separate branches to get there, even if those branches are identical. This wastes space.

A decision jungle changes this by allowing these separate branches to merge back into a single path. Imagine you are looking for a specific book in a library. In a standard tree, every search query forces you down a unique aisle. If two different queries lead to the same section, you still walk two different aisles. In a jungle, once your search reaches that common section, the paths merge. You stop walking distinct routes and share the rest of the journey.

Let’s look at a concrete example. Suppose you are deciding whether to bring an umbrella based on weather data. A standard tree might have a branch for 'High Humidity + Low Wind' and another for 'High Humidity + High Wind'. If both lead to the decision 'Bring Umbrella', the tree stores that final decision twice. In a jungle, these two paths merge before the final step. You only store the 'Bring Umbrella' logic once. This means you use fewer nodes (the points where decisions are made) to represent the same information. The result is a model that takes up less memory on your device and runs faster, because it has fewer pieces of data to look at as it processes new input.

The trade-off is complexity in building the model. You have to decide exactly where these paths can safely merge without losing accuracy. But when you are working with tight storage limits, a jungle allows you to fit complex logic into small spaces that would otherwise overflow.


## An analogy

Imagine a city built on a grid of one-way streets. A standard decision tree is like a city where every trip from the suburbs to downtown requires a completely private road. Even if two neighborhoods are next to each other and both need to go to the same plaza, they get two separate, never-merging roads. This is incredibly wasteful of land and construction materials.

A decision jungle is like a city with a smart highway system. Cars from different neighborhoods can travel on their own local streets, but once they are close to the center, they merge onto shared highways. Everyone uses the same main road for the last part of the trip. This saves space and makes travel more efficient because there are fewer roads to build and maintain.

The catch is planning. You cannot let traffic flow backward or in circles; you must plan the merges carefully so that a car coming from the north doesn't accidentally end up going south when it meant to go east. In a decision jungle, this means once paths merge, they move forward toward a conclusion and never loop back, ensuring the logic remains clear and efficient.


## Definition

A decision jungle is a machine learning structure that allows multiple decision paths to merge into shared nodes, reducing the total amount of data needed to represent the model compared to separate branches.


## How it connects

This builds on your understanding of Decision Trees, which are simple models that split data step-by-step. A jungle modifies this by letting those steps share common endings, which is a form of Model Compression, where you reduce the size of a model without losing its predictive power. This is distinct from Ensemble Methods, which combine many separate models to improve accuracy, whereas a jungle combines parts of one logical structure to save space.


## The mathematics

A decision jungle is a way of organizing decisions that looks like a map with many paths leading to the same destination. To understand the math, we first need to define what a "graph" is in this context.

A graph is made of two things: nodes and edges. A node is a point where something happens (like asking a question). An edge is a line connecting two nodes, showing which question comes next. In a standard decision tree, the structure is strict: every node has exactly one parent (the question that led to it), except for the very first node, called the root. This creates a single path from the start to any end point.

A decision jungle relaxes this rule. It allows a node to have more than one parent. This means you can reach the same question from different starting points. Because there are multiple paths, we must ensure there are no loops. A loop would mean you could follow the edges forever without ending. Mathematically, we call a graph with no loops a "directed acyclic graph" (DAG). The word "directed" means the edges have arrows pointing one way. The word "acyclic" means you cannot start at a node and follow the arrows to return to that same node.

The main mathematical benefit of a jungle is saving space. In a forest of separate trees, if two trees ask the same question and lead to the same future steps, that information is stored twice. In a jungle, we store it only once. We call this "merging" or "sharing" subtrees.

To calculate the savings, we compare the total number of nodes in a forest to the total number of nodes in the corresponding jungle.

Let N_F be the total count of all nodes if we kept every tree separate.
Let N_J be the total count of all unique nodes in the merged jungle.
The saving S is the difference: S = N_F minus N_J.

How do we find N_J? We start with the forest count N_F. Then, for every group of identical subtrees that we merge, we remove the duplicates. If we merge a group of k identical subtrees, each having s nodes, we keep one copy and discard k minus 1 copies. The number of nodes removed is (k minus 1) times s.

So, if we have several groups of merged subtrees, we subtract the saved nodes from the original total:
N_J = N_F minus the sum of (s_i minus 1) for each merged group i.

The goal of the algorithm is to find a graph structure that makes S as large as possible (saving the most space) while still making accurate predictions. The training process uses complex calculus to decide which questions to ask, but the structural math of the jungle itself is just counting nodes and subtracting the duplicates.


## A worked example

Let us walk through a specific example with real numbers to see how the node count changes.

Imagine we are building a model to decide whether to approve a loan. We have three features: income, debt, and tenure (how long you have been at your job). The outcome is binary: approve or reject.

First, consider a single decision tree. It might look like this:
1. Root Node: Is income greater than 5? 
   - If yes, go to Node A.
   - If no, go to Node B.
2. Node A: Is debt greater than 3?
   - If yes, go to Leaf 1 (Approve).
   - If no, go to Leaf 2 (Approve).
3. Node B: Is tenure greater than 2?
   - If yes, go to Leaf 3 (Approve).
   - If no, go to Leaf 4 (Reject).

Let us count the nodes in this single tree:
- 1 Root node.
- 2 Internal nodes (Node A and Node B) where questions are asked.
- 4 Leaf nodes (Leaf 1 through Leaf 4) where the final decision is made.
Total nodes in one tree = 1 + 2 + 4 = 7 nodes.

Now, imagine we have a forest of two such trees. In a standard forest, Tree 2 is built separately. Even if it asks similar questions later, it has its own copies of every node.
Step 1: Calculate the forest size (N_F).
We have 2 trees. Each tree has 7 nodes.
N_F = 2 * 7 = 14 nodes.

Now, let us build a jungle from these two trees. Suppose Tree 2 starts with a different first question (e.g., "Is debt greater than 2?"). However, after some steps, both Tree 1 and Tree 2 arrive at the exact same situation: they need to check if tenure is greater than 2, and the subsequent leaves are identical.

In a jungle, we do not create new nodes for this shared part. We point both trees to the existing "tenure" node.

Let us count the nodes in the jungle (N_J):
- Tree 1 has its own root and its own internal node (checking debt). That is 2 nodes.
- Tree 2 has its own root and its own internal node (checking income, perhaps). That is 2 nodes.
- The shared part consists of the "tenure" node and the two leaves attached to it. This is a single subtree with 3 nodes (1 tenure node + 2 leaves).

Total nodes in the jungle = 2 (from Tree 1 unique start) + 2 (from Tree 2 unique start) + 3 (shared end) = 7 nodes.
N_J = 7 nodes.

Step 2: Calculate the saving (S).
S = N_F - N_J
S = 14 - 7 = 7 nodes saved.

Step 3: Calculate the percentage reduction.
Reduction = S / N_F
Reduction = 7 / 14 = 0.5 or 50%.

Result: In this example, the jungle uses half as many nodes as the forest. The saving comes entirely from merging the identical subtree at the end. In real-world applications, the amount of sharing varies, but reductions of 20–50% are common when many trees in a forest have similar structures.


## From scratch, in Python

```python
class Node:
    def __init__(self, feature=None, threshold=None, left=None, right=None, leaf=False, value=None):
        self.feature = feature
        self.threshold = threshold
        self.left = left
        self.right = right
        self.leaf = leaf
        self.value = value

def predict(node, sample):
    if node.leaf:
        return node.value
    if sample[node.feature] <= node.threshold:
        return predict(node.left, sample)
    else:
        return predict(node.right, sample)

def build_forest(data, labels, n_trees):
    # Simplified: train each tree on a bootstrap sample
    forest = []
    for i in range(n_trees):
        indices = [j % len(data) for j in range(i, len(data) + i)]  # pretend bootstrap
        subset_data = [data[j] for j in indices]
        subset_labels = [labels[j] for j in indices]
        tree = build_tree(subset_data, subset_labels)
        forest.append(tree)
    return forest

def build_jungle(data, labels, n_trees):
    # First, build a forest
    trees = build_forest(data, labels, n_trees)
    # Then, merge identical subtrees (in this simple example, we just share the root of a common subtree)
    # For simplicity, we will not modify the structure but show the concept of sharing by returning a dict of roots.
    roots = {}
    for i, t in enumerate(trees):
        roots[i] = t
    # Merge: if two trees have the same structure after a certain point, we could replace one branch with a pointer
    # Here, we artificially merge two trees if their right subtrees are identical
    # (in real code this would be done with hashing and a DAG builder)
    # For the purpose of this demonstration, we return the forest as-is and note the potential savings.
    return roots, None  # None because we did not actually merge for simplicity

def build_tree(data, labels):
    # Placeholder for a real tree learner (e.g., ID3 or CART)
    # For this demo, we always return a single leaf with the majority label
    from collections import Counter
    counter = Counter(labels)
    majority = counter.most_common(1)[0][0]
    return Node(leaf=True, value=majority)
```


## What the code does

The code defines a simple Node class for a decision tree. Each node either is a leaf (with a predicted value) or has a feature index and threshold to split on. The predict function recursively walks down the tree, following the branch based on the sample's feature value. The build_forest function creates a list of trees, each trained on a different bootstrap sample (simulated by a simple rotation). The build_jungle function is a stub that shows the concept of merging: in a real jungle, you would traverse the trees and identify shared subtrees, then point both parents to the same subtree instead of having separate copies. The from-scratch code here does not perform that merge to keep it runnable and understandable, but the walkthrough explains how a merge would be implemented using hashing of subtrees and a DAG structure.


## Common misconceptions

**Jungles are just trees with shared nodes** — A jungle is exactly the same as a forest, except I can save memory by putting two trees side by side.

A forest keeps trees completely independent; a jungle is a single DAG where nodes can have multiple parents, so the structure is fundamentally different. In fact: A jungle is a single graph that is trained as a whole, allowing paths to merge. You cannot convert a forest to a jungle by simply pointing to shared nodes without re-training or careful validation.

**Jungles always outperform forests** — Since jungles use less memory, they must be more accurate than forests.

Merging reduces the model's capacity, which could hurt accuracy if the merged regions are not truly identical in behaviour. In fact: Jungles are a compression technique. They trade a small amount of accuracy for a large reduction in memory, and in many cases the accuracy loss is negligible, but it is not guaranteed to be zero.

**Jungles are a type of neural network** — Because jungles use directed acyclic graphs, they are basically neural networks.

Neural networks have weighted connections and learn continuous functions through backpropagation; jungles are based on threshold splits and do not use weights or gradients in the same way. In fact: Jungles belong to the decision tree family. The DAG structure is the only similarity to neural networks; the learning algorithm is completely different.

**Merging paths always reduces memory** — If I merge any two identical subtrees, I will always save memory.

Merging requires storing pointers and possibly extra bookkeeping. In small graphs, the overhead can outweigh the savings. In fact: Merging saves memory only if the size of the merged subtree is larger than the pointer overhead. In practice, the savings are only meaningful for substantial subtrees.


## In short

Decision jungles are a way to make decision forests more memory-efficient by allowing paths to merge. They are a directed acyclic graph rather than a collection of independent trees. The key benefit is reduced model size, which is crucial for on-device deployment. The trade-off is more complex training and a potential small loss in accuracy. For situations where memory is tight, a jungle can be the difference between a model that fits and one that does not.


## Where this is used

Decision jungles are primarily used in situations where model size is a hard constraint, such as on-device machine learning. In large language models (LLMs), decision jungles are not used directly, but the principle of sharing computation is relevant: models like transformers share weights across all tokens, which is a form of reuse. In small language models running on phones, decision jungles could be used for lightweight classification tasks (e.g., intent recognition or word detection) instead of a full neural network. In retrieval-augmented generation (RAG), decision jungles are not common, but they could be used to route queries to different retrievers with a compact classifier. In agentic systems, jungles could be used for decision-making about which tool to call, when the model needs to be small and fast. In everyday applied machine learning, jungles are used when deploying models to embedded devices like microcontrollers, where memory is measured in kilobytes. A developer would use a library like the Decision Jungle implementation in Microsoft's ML.NET or write a custom graph-based model. The trade-off is that jungle training is more complex and less mature than gradient boosting, so practitioners often start with a forest and then compress it if necessary. The breakage risk: if you naively merge subtrees that are not truly identical, you will introduce errors in predictions, so the merging must be done carefully with validation.


---

Read this on the web: https://basavapurushottam.com/learn-ai/decision-jungles
