# Kolmogorov-Arnold Networks

Networks placing learnable activation functions on edges rather than fixed activations on nodes, inspired by the Kolmogorov-Arnold representation theorem.

**Rests on:** Activation Functions, Forward Propagation

*Deep Learning · Deep Learning*


## Why learn this

This gives you a way to understand neural networks that behaves more like the math you might recognize from school, rather than an opaque black box. Most standard models treat data as a pile of numbers to be multiplied and summed, making it hard to see *why* a decision was made. Kolmogorov-Arnold Networks (KANs) change the structure so that the relationships between variables are learned directly as continuous curves. This matters because it allows you to inspect exactly how each input influences the output—for example, seeing clearly if the relationship is linear, exponential, or periodic—making the model transparent and interpretable by design. While standard networks hide their logic inside millions of tiny weights, KANs expose the actual mathematical form connecting the inputs to the result.


## The intuition

Imagine you are trying to map the relationship between two things, such as the weight of a box and the effort needed to lift it. In a traditional approach, you might use a rigid, step-by-step filter that applies the same rule everywhere: 'if it’s light, add 10 units of force; if it’s heavy, add 20.' This is like a fixed machine. A KAN works differently. Instead of a fixed rule, imagine a flexible rubber band stretched between the weight and the effort. As you learn from examples, this rubber band stretches and curves to fit the actual data points perfectly. One input variable might be connected to the output by a straight line (linear relationship), while another is connected by a steep curve (exponential relationship). The network doesn't guess the shape; it learns the exact curve that fits the pattern in your data. This means you can look at the final model and read the actual function that connects cause and effect, rather than just seeing a list of numbers.


## An analogy

Think of a traditional neural network as a musician playing a piece of sheet music where every note is fixed in advance; they can only adjust how loudly or softly they play (the volume) but cannot change the notes themselves. A KAN is like a jazz improviser who can change the melody itself. The traditional musician must rely on having many complex, fixed scales to cover all possible songs. The improviser can reshape the song in real-time to match the listener's mood perfectly. In both cases, the performance is built from simple elements added together, but the KAN’s ability to reshape its own 'melody' for each input makes it much more efficient at capturing unique patterns without needing layers upon layers of repetition. The analogy breaks down because a musician has conscious intent, while a network learns through mathematical optimization.


## Definition

A Kolmogorov-Arnold Network is a type of neural network that replaces fixed mathematical operations with learnable, flexible curves on each connection, allowing the model to discover the exact functional relationship between inputs and outputs during training.


## How it connects

You can compare this to Activation Functions, which are the static rules (like 'clamp negative values to zero') used in traditional networks. It also relates to Universal Approximation Theorems, which state that any complex curve can be built from simpler ones—a principle KANs apply by learning those simpler curves directly rather than using fixed building blocks.


## The mathematics

To understand Kolmogorov-Arnold Networks (KANs), we must first look at how a standard neural network works, and then see what KAN changes.

**1. The Standard Neural Network Layer**
A standard layer takes several input numbers and produces one output number. Let us say the inputs are x₁, x₂, ..., xₙ. The calculation happens in two steps:

Step 1: Weighted Sum. Each input xᵢ is multiplied by a specific number called a weight, wᵢⱼ. These products are added together, and a constant number called a bias, bⱼ, is added to the total.
The formula for this sum is:
Sum = (x₁ × w₁ⱼ) + (x₂ × w₂ⱼ) + ... + (xₙ × wₙⱼ) + bⱼ

Step 2: Activation. The result of the Sum is passed through a fixed 'squashing' function, usually called sigma (σ). This function takes any number and squeezes it into a specific range, often between 0 and 1. For example, if the sum is 5, σ(5) might be 0.99; if the sum is -2, σ(-2) might be 0.12.
The final output yⱼ is:
yⱼ = σ(Sum)

The key feature here is that the weights (w) and bias (b) are simple numbers. The shape of the squashing function σ is fixed in advance and does not change during learning.

**2. What a KAN Layer Does Differently**
A Kolmogorov-Arnold Network removes the multiplication by weights and the fixed squashing function. Instead, it uses a 'learnable curve' for each input.

Imagine you have one input, x₁. In a KAN, you do not multiply x₁ by a weight. Instead, you pass x₁ through a specific curve, which we will call φ (pronounced 'phi'). This curve is a function that takes x₁ as input and returns a new number.

If you have multiple inputs (x₁, x₂, ..., xₙ), each one goes through its own learnable curve. Let's say the curve for input i is called φᵢⱼ. The subscript j indicates which output neuron we are calculating for.

Step 1: Apply Curves. For each input xᵢ, calculate the value of the curve at that point:
Value₁ = φ₁ⱼ(x₁)
Value₂ = φ₂ⱼ(x₂)
...
Valueₙ = φₙⱼ(xₙ)

Step 2: Sum the Results. Add all these values together to get the output yⱼ:
yⱼ = Value₁ + Value₂ + ... + Valueₙ
Or written as a sum:
yⱼ = Σᵢ φᵢⱼ(xᵢ)
Here, the symbol Σ (sigma) means 'sum of'. It tells us to add up the term φᵢⱼ(xᵢ) for every i from 1 to n.

**3. Why This Works: The Kolmogorov-Arnold Representation Theorem**
The name 'Kolmogorov-Arnold' comes from a mathematical theorem proved by Andrey Kolmogorov and Vladimir Arnold. The theorem states that any continuous function of many variables can be built using only sums and functions of one variable.

In simpler terms: if you have a complex relationship between many inputs (like x₁, x₂, etc.) that produces an output, you can break that relationship down. You can transform each input individually using a specific curve, and then just add those transformed values together to get the correct output.

A KAN takes this theorem literally. It assumes that the network's job is to find the right shapes for these individual curves (φ) so that when they are added up, the result matches the desired answer. The 'learning' process in a KAN is simply finding the best shape for each curve φ.

**4. Comparing the Two Approaches**
Standard Network: Multiply input by weight → Add biases → Apply fixed squashing function.
KAN: Apply learnable curve to input → Sum the results.

The KAN approach replaces the rigid arithmetic of multiplication with flexible geometry (curves). This allows the network to capture complex patterns without needing as many layers or neurons, because each curve can be very detailed and 'wiggly'.


## A worked example

Let us build a tiny KAN with just one input, x. We will define one learnable curve, φ, for this single input.

To make the curve easy to calculate, we will define it using three control points. A control point is a specific location (x-value, y-value) that the curve must pass through. We connect these points with straight lines. This creates a 'piecewise linear' curve.

**The Control Points:**
Point 1: At x = 0, the curve height is 1. So, φ(0) = 1.
Point 2: At x = 1, the curve height is 3. So, φ(1) = 3.
Point 3: At x = 2, the curve height is 5. So, φ(2) = 5.

The curve looks like this:
- From x=0 to x=1, it is a straight line going from height 1 to height 3.
- From x=1 to x=2, it is a straight line going from height 3 to height 5.

**Calculation Example 1: Input x = 0.5**
We need to find φ(0.5). The value 0.5 lies between our first two control points (x=0 and x=1).

Step 1: Find the change in height over this segment.
The height at x=1 is 3. The height at x=0 is 1. The total rise is 3 - 1 = 2.

Step 2: Find how far along the segment we are.
The input x=0.5 is exactly halfway between 0 and 1. So we have traveled half the distance.

Step 3: Calculate the height.
Start at the bottom height (1) and add half of the rise (2).
Half of 2 is 1.
Height = 1 + 1 = 2.
So, φ(0.5) = 2.

**Calculation Example 2: Input x = 1.5**
We need to find φ(1.5). The value 1.5 lies between our second and third control points (x=1 and x=2).

Step 1: Find the change in height over this segment.
The height at x=2 is 5. The height at x=1 is 3. The total rise is 5 - 3 = 2.

Step 2: Find how far along the segment we are.
The input x=1.5 is exactly halfway between 1 and 2.

Step 3: Calculate the height.
Start at the bottom height of this segment (which is 3) and add half of the rise (2).
Half of 2 is 1.
Height = 3 + 1 = 4.
So, φ(1.5) = 4.

**Calculation Example 3: Input x = 0.25**
We need to find φ(0.25). This value is between x=0 and x=1.

Step 1: The segment goes from height 1 (at x=0) to height 3 (at x=1).
The total rise is 2 over a distance of 1 unit in x.
This means the slope is 2. For every 1 unit you move right, you go up 2 units.

Step 2: Calculate the rise for x=0.25.
Since we moved 0.25 units to the right from x=0, we go up 0.25 × 2 = 0.5 units.

Step 3: Add to the starting height.
Starting height is 1.
Final height = 1 + 0.5 = 1.5.
So, φ(0.25) = 1.5.

**Summary of the KAN Output**
In this simple one-input KAN, the output y is just φ(x).
If x=0.5, y=2.
If x=1.5, y=4.
If x=0.25, y=1.5.

The 'learning' process would involve changing the heights of the control points (1, 3, 5) to different numbers until the curve φ produces outputs that help the network solve its specific problem.


## From scratch, in Python

```python
This code shows a single KAN layer with one input and one output.

# No imports needed for the core logic, just plain Python.

# Step 1: Represent the learnable function as a list of control points.
# Each control point is a tuple: (x_position, y_value).
# We start with three points: at x=0, y=1; at x=1, y=3; at x=2, y=5.
control_points = [(0, 1), (1, 3), (2, 5)]

# Step 2: Define a function that evaluates our learnable curve at any x.
def evaluate_curve(x, points):
    """Find the y-value of a piecewise-linear curve at x."""
    # If x is off the left edge, return the first y-value.
    if x < points[0][0]:
        return points[0][1]
    # If x is off the right edge, return the last y-value.
    if x > points[-1][0]:
        return points[-1][1]
    # Otherwise, find the two points x falls between.
    for i in range(len(points) - 1):
        x1, y1 = points[i]
        x2, y2 = points[i + 1]
        if x1 <= x <= x2:
            # Interpolate: weight the two y-values by how close x is to each x.
            # If x is exactly x1, we want y1. If x is exactly x2, we want y2.
            fraction = (x - x1) / (x2 - x1)
            return y1 + fraction * (y2 - y1)

# Test it with different x values.
print(evaluate_curve(0.5, control_points))  # Should print 2.0
print(evaluate_curve(1.5, control_points))  # Should print 4.0

# Step 3: This is the 'layer'. For one input, one output.
# It just evaluates the curve on the input.
input_x = 0.25
output_y = evaluate_curve(input_x, control_points)
print(output_y)  # Should print 1.5
```


## The same thing, with a framework

```python
A real KAN implementation would use something like PyTorch. Here's the idea:

import torch
import torch.nn as nn

class KANLayer(nn.Module):
    def __init__(self, in_features, out_features, grid_size=5):
        super().__init__()
        # Create a grid of learnable parameters for each edge.
        # This is a tensor of shape (out_features, in_features, grid_size).
        self.weights = nn.Parameter(torch.randn(out_features, in_features, grid_size))
        self.grid = torch.linspace(-1, 1, grid_size)

    def forward(self, x):
        # For simplicity, this just evaluates a linear combination of
        # the grid points; a real KAN uses splines or B-splines (see notes).
        batch = x.shape[0]
        x = x.unsqueeze(1).unsqueeze(2)  # shape: batch, 1, 1
        # Simplified: treat each weight as a coefficient multiplied by the input.
        # This is NOT a real KAN; it just shows the tensor structure.
        out = (self.weights * x).sum(-1)  # sum over grid
        return out.squeeze(1)
```


## What the code does

**From scratch code:**

We use **lists** and **tuples**. A **tuple** is like a list but you cannot change it — `(0, 1)` is a tuple holding two numbers. Our `control_points` is a **list** of these tuples, keeping them in order: `[(0, 1), (1, 3), (2, 5)]`. Each tuple holds the x-position and y-value of a point on our curve.

The `evaluate_curve` function takes two inputs: a number `x` and a list of points. A **function** is a named block of reusable code. We first check if `x` is outside the range of our points; if so, we return the edge value. Then we use a **for loop** to look at each pair of neighbouring points. `for i in range(len(points) - 1)` runs the block for `i` equal to 0, then 1. This is a **loop**, a way to repeat instructions. Inside the loop, we unpack the tuple: `x1, y1 = points[i]` pulls the two values out. We check if our `x` is between `x1` and `x2`. If it is, we interpolate. The line `fraction = (x - x1) / (x2 - x1)` gives a number between 0 and 1: 0 when `x` is at `x1`, 1 when at `x2`. Then `y1 + fraction * (y2 - y1)` gives a weighted mix of the two y-values. This is exactly the arithmetic from the mathematics section.

**Framework code:**

This is a rough sketch of how a KAN layer might look in PyTorch, a popular library. It uses **tensors** — like lists but specialised for maths on GPUs. The `self.weights` is a tensor with three dimensions: the output feature, the input feature, and the grid points. The forward pass computes a simplified output by multiplying the input by the weights and summing over the grid. A real KAN replaces this with spline interpolation, which is a more powerful version of our piecewise-linear curve.


## Common misconceptions

**"There's no bias term in a KAN."** — I heard KANs don't have biases, so they must be a different kind of network with no way to shift the activation.

The learnable activation function on each edge can itself shift up or down along the y-axis. That shift absorbs the role that a bias would normally play. The connection is learned rather than stored as a separate number. In fact: The bias is effectively absorbed into the learnable activation function on each edge. The function can learn to be offset, so a separate bias parameter is not needed.

**"KANs use one activation per node, like normal networks."** — KANs are just like standard networks but with better activations on each neuron.

The defining difference is that KANs place learnable functions on the *edges* (connections), not on the *nodes* (neurons). This is central and changes what is learned. In fact: In a KAN, the learnable activation function is on the connection between two neurons. A normal network has a fixed activation on each neuron and only learns the weights on the connections.

**"KANs are obviously better, the hype must be true."** — KANs are the successor to Transformers, so all new AI systems will use KANs.

KANs as a research direction are far from a settled default. They have serious trade-offs: slower training, harder optimisation, and less mature software. They are promising for specific niches like interpretability and parameter efficiency, not a universal replacement. In fact: KANs are an active research direction with real drawbacks. They are compelling where parameter efficiency and interpretability matter, but standard networks remain the go-to for many tasks.


## In short

KANs flip the traditional network recipe: instead of a fixed activation function on each node and learnable weights on the edges, they put a learnable one-dimensional function on each edge and simply sum the results. This approach, inspired by the Kolmogorov-Arnold theorem, promises better parameter efficiency and interpretability. They are a promising research direction, not a settled default, and understanding them means understanding what an activation function does inside a network.


## Where this is used

KANs are an active research area, not a settled industry standard. They are not yet the engine inside products like ChatGPT, search, or recommendation systems. That said, they appear in several places:

**Large Language Models (LLMs):** Today's transformers use fixed activation functions (like GELU) inside their feed-forward networks. Researchers are actively exploring whether replacing the MLP blocks with KAN layers can make LLMs more parameter-efficient. The idea is that KANs can learn more complex internal representations with fewer parameters, because the activation functions are flexible. A developer researching this would take a trained transformer and swap the MLP blocks for KAN layers, then fine-tune. The experiment often shows the model needs fewer parameters to match performance, but training is slower and the KAN layers can be harder to train stably.

**Small Models on Devices (TinyML):** This is where KANs' parameter efficiency could shine. A small model — for keyword spotting on a phone or a sensor in a factory — has a strict memory budget. If a KAN can match the accuracy of a traditional network with fewer parameters, it fits in the same memory. The application is the same, but the model's weights are replaced by the coefficients of the learnable curves. A developer would use a KAN library to build a tiny speech model, quantise it (reduce the precision of the numbers) to fit in a few hundred kilobytes, and verify it still runs in real time on the device's small processor.

**Retrieval-Augmented Generation (RAG):** The core of RAG is a similarity search over embeddings — lists of numbers representing text. This doesn't directly use KANs. However, KANs have been proposed for *reranking*: after a fast search returns candidate documents, a reranker re-scores them to pick the best. Traditional rerankers are small transformers. A KAN-based reranker could, in principle, learn a richer scoring function between the query and document embeddings, capturing more complex relevance patterns than a dot product. The practical implementation would use a KAN layer as the final scoring head, taking the concatenated query and document embeddings as input and producing a relevance score.

**Agentic Systems (Planning, Tools, Loops):** Agents don't use KANs directly. The neural networks inside agents handle perception and language. KANs don't change the planning loop or the tool-calling logic. However, if KANs make the internal models more efficient and interpretable, an agent could understand *why* it made a decision — because each KAN edge is a visible, learnable function. This transparency is motivating research into using KANs for safe, explainable decision modules inside autonomous systems, but this is speculative and not yet production practice.

**Everyday Applied Work (Forecasting, Fraud, Recommendation):** In tabular data problems — fraud detection on a credit-card transaction, demand forecasting from historical sales — KANs are a serious candidate. A developer struggling to get a gradient-boosted tree (a traditional non-neural method) to pick up a subtle nonlinear pattern might try a KAN. The win is two-fold: parameter efficiency on small datasets and interpretability. For fraud detection, a bank can inspect the learned curve on 'transaction amount' and see exactly how the model treats amounts above ₹50,000 — it becomes starkly visible when the curve steepens. In practice, a developer would import a KAN library, feed their normalised features in, train with standard optimisers, and monitor validation loss against their baseline model.

**What breaks when it's done badly:** Training KANs is substantially slower because of the complex learnable functions. Using KANs for a vision task where a standard CNN already works wastes compute. And because they're new, the APIs are less stable — a model built on a research library today may break when the library updates.


---

Read this on the web: https://basavapurushottam.com/learn-ai/kolmogorov-arnold-networks
