# Fourier Analysis of Networks

Analysing what frequencies a network represents and learns, revealing systematic preferences in what it fits first.

**Rests on:** Convolutional Layers, Forward Propagation

*Deep Learning · Convolutional Networks*


## Why learn this

You want to understand why your models sometimes seem stuck or why they capture broad trends before fine details. This knowledge lets you diagnose training issues by looking at what the network is actually learning: simple, smooth structures first, and complex, jagged details last. It explains why a model might look perfect on general data but fail on subtle variations, and it gives you a way to predict how changes in your setup will affect what patterns the network picks up.


## The intuition

Imagine a photograph of a landscape. You can describe this image in two ways. First, by looking at the pixels directly: the green of the trees, the blue of the sky, the sharp line where the horizon meets the sea. This is viewing the data as it sits on the grid of pixels. Second, you can break the image down into its constituent waves. A smooth gradient in the sky is a low-frequency wave—a slow, gentle change in color. The sharp edge of a tree branch is a high-frequency wave—a rapid, sharp change.

Fourier analysis is the tool that performs this breakdown. It tells you which 'waves' make up the image and how strong each one is.

When a neural network learns to recognize patterns in an image, it does not do so randomly. It has a predictable order of operations. In the early stages of training, the network focuses almost entirely on low-frequency components. It learns the broad shapes: the general outline of a face, the large blocks of color in a room, the smooth contours of a landscape. These are the 'easy' parts of the data because they involve gradual changes that are stable and consistent.

Only after it has mastered these broad structures does the network begin to learn high-frequency components. These are the fine details: the texture of skin, the individual hairs on an animal, the sharp edge of a leaf against a sky. Learning these details is harder because they change rapidly over small distances, making them noisy and unstable for the network to grasp initially.

This ordered learning—smooth patterns first, fine details later—is called spectral bias. It explains why networks are good at generalization: by capturing the fundamental structure before worrying about noise, they build a robust understanding of what they are looking at. It also explains why networks can be fooled: if you add tiny, high-frequency noise to an image that is imperceptible to human eyes (like changing pixel values by 1%), the network might ignore the overall shape it learned first and get confused by the jagged new detail.


## An analogy

Think of learning to paint a realistic portrait. On your first day, you do not start by trying to capture the individual pores on the skin or the glint of light in an eye. You start with the broad shapes: the oval of the face, the placement of the eyes and nose, the general shadows and highlights. These are the low-frequency elements—the large, smooth areas of color and tone.

Only after you have blocked in these broad shapes correctly do you move to the fine details. You start adding the eyelashes, the wrinkles around the eyes, and the subtle skin texture. These are the high-frequency elements—rapid changes in value over very short distances.

A neural network works exactly this way. Its architecture and training process naturally push it to lock onto the broad, low-frequency 'sketch' of the data first. The mathematics governing how it updates its knowledge creates a landscape where moving toward those broad patterns is easy and direct, while navigating the jagged, noisy details is difficult and slow. The network isn't choosing this order; it is a physical consequence of how the learning algorithm operates. It is easier to find the bottom of a wide valley than to pick your way through a narrow, rocky canyon.

A caveat: unlike painting, the network cannot consciously decide to skip steps; the bias is hardwired into the math of the update rule.


## Definition

Fourier analysis of networks is the study of how neural networks process information by separating data into low-frequency (smooth, broad) and high-frequency (fine, detailed) components, revealing that networks systematically learn the smooth patterns before the fine details due to the mathematical properties of their training process.


## How it connects

This concept extends your understanding of Convolutional Networks. In convolutional layers, you apply filters (small grids of numbers) across an image in what is called the spatial domain—the original pixel grid. Fourier analysis offers a different view: the frequency domain. Here, you ask which waves pass through the filter and which are blocked. It also connects to CNN Visualization. When you saw that early layers detect edges and colors, you were seeing low-frequency patterns. Later layers combining these into complex objects involves higher frequencies.


## The mathematics

To understand how Convolutional Networks work, we need to look at them through two different lenses: the lens of space (where pixels live) and the lens of frequency (where waves live). This connection is called Fourier Analysis.

Imagine a sound wave. A complex sound, like a chord on a piano, is made of several pure tones mixed together. Fourier Analysis is the tool that lets us break that complex sound down into its individual pure tones. It tells us: "How much of tone A is here? How much of tone B is here?"

In an image, instead of sound waves, we have patterns of brightness. A sharp edge in an image is like a high-pitched note (a high frequency). A smooth gradient is like a low-pitched note (a low frequency).

The Convolutional Network performs an operation called **convolution**. In the spatial lens, convolution means taking a small grid of numbers (the filter) and sliding it over the image. At each position, you multiply the filter numbers by the pixel numbers underneath them and add up the results.

This is computationally heavy if done naively for large images. However, there is a powerful mathematical shortcut called the **Convolution Theorem**.

The theorem states:
If you take the Fourier Transform of an image (breaking it into frequencies) AND take the Fourier Transform of the filter (breaking it into frequencies), then multiplying those two frequency lists together gives you exactly the same result as doing the sliding multiplication in space, but only if you transform the result back.

In simpler terms:
Convolution in Space = Multiplication in Frequency.

This is not just a curiosity; it explains how the network learns. When we train a network using **gradient descent** (a method that adjusts the filter numbers to reduce error), the math shows that low frequencies change the error in a smooth, predictable way. High frequencies change the error in a jagged, unpredictable way.

Because gradient descent follows the slope of the error, it finds clear paths for low frequencies but gets lost in the noise of high frequencies. This is why networks tend to learn simple, broad patterns (low frequencies) before they learn fine details (high frequencies).

Let us define the symbols we will use:
- Let f be the input signal (the image or data).
- Let g be the filter (the pattern we are looking for).
- Let * denote convolution. This means "slide and multiply-sum".
- Let F{} denote the Fourier Transform. This means "break into frequencies".
- Let · denote element-wise multiplication. This means "multiply corresponding items in two lists".

The Convolution Theorem is written as:
F{ f * g } = F{ f } · F{ g }

Read aloud: "The Fourier Transform of (f convolved with g) is equal to the Fourier Transform of f multiplied by the Fourier Transform of g."

In a neural network, if x is the input layer, w is the filter weights, and y is the output:
Y(nu) = X(nu) · W(nu)

Here, nu (nu) represents a specific frequency index. This equation says: The output frequency content is just the input frequency content scaled by the filter's frequency content.


## A worked example

Let us walk through a tiny example to see how convolution works in space, and why the frequency view matters.

We have a 1D signal (a row of 4 pixels): x = [1, 0, -1, 0].
We have a filter (a small window) w = [0.5, 0.5]. This filter calculates the average of two adjacent pixels.

Step 1: Perform convolution in space.
Convolution slides the filter over the signal. We multiply the filter values by the signal values at each position and sum them.

Position 1 (covering x[0] and x[1]):
Multiply: 1 * 0.5 = 0.5
Multiply: 0 * 0.5 = 0
Sum: 0.5 + 0 = 0.5
Output value 1 is 0.5.

Position 2 (covering x[1] and x[2]):
Multiply: 0 * 0.5 = 0
Multiply: -1 * 0.5 = -0.5
Sum: 0 + (-0.5) = -0.5
Output value 2 is -0.5.

Position 3 (covering x[2] and x[3]):
Multiply: -1 * 0.5 = -0.5
Multiply: 0 * 0.5 = 0
Sum: -0.5 + 0 = -0.5
Output value 3 is -0.5.

The final output signal in space is [0.5, -0.5, -0.5].

Step 2: Understand the frequency behavior.
Instead of doing this sliding math every time, we can look at what the filter does to frequencies.
The filter w = [0.5, 0.5] is a "low-pass filter." This means it keeps low-frequency patterns (smooth changes) and reduces high-frequency patterns (sharp jumps).

If we were to compute the Fourier Transform of x and w:
- The Fourier Transform of w shows that it has strong power at low frequencies and weak power at high frequencies.
- When we multiply F{x} by F{w}, we are scaling the input's frequencies by the filter's strength.
- Low frequencies in x are multiplied by a large number (close to 1), so they stay strong.
- High frequencies in x are multiplied by a small number (close to 0), so they become weak.

This confirms that the sliding operation [0.5, -0.5, -0.5] is mathematically equivalent to scaling the frequencies of the input. The network learns by adjusting w. If it wants to detect edges (high frequency), it must change w to have high values in its own frequency representation. But as noted, gradient descent finds it easier to adjust w for low frequencies first.


## From scratch, in Python

```python
import math

# A tiny 1D signal: 4 values
def convolve_1d(signal, kernel):
    # signal: list of numbers, kernel: list of numbers
    # output length = len(signal) - len(kernel) + 1, without padding
    output_length = len(signal) - len(kernel) + 1
    output = []
    for i in range(output_length):
        # dot product of kernel with the overlapping part of signal
        total = 0
        for j in range(len(kernel)):
            total += signal[i + j] * kernel[j]
        output.append(total)
    return output

# A discrete Fourier transform (simplified for real signals, just for demonstration)
def dft(signal):
    N = len(signal)
    real = [0.0] * N
    imag = [0.0] * N
    for k in range(N):  # frequency index
        for n in range(N):  # time index
            angle = 2 * math.pi * k * n / N
            real[k] += signal[n] * math.cos(angle)
            imag[k] -= signal[n] * math.sin(angle)
    return real, imag

# Test with our signal and kernel
signal = [1, 0, -1, 0]
kernel = [0.5, 0.5]

print("Convolution (spatial):", convolve_1d(signal, kernel))

# Now compute the DFT of both
real_sig, imag_sig = dft(signal)
real_ker, imag_ker = dft(kernel)  # kernel length is 2, but DFT expects same length; we pad with zeros
# For simplicity, we'll pad the kernel to length 4
kernel_padded = kernel + [0, 0]
real_ker, imag_ker = dft(kernel_padded)

# Multiply in frequency domain (complex multiplication)
# (a+bi)*(c+di) = (ac-bd) + (ad+bc)i
real_prod = [real_sig[i]*real_ker[i] - imag_sig[i]*imag_ker[i] for i in range(4)]
imag_prod = [real_sig[i]*imag_ker[i] + imag_sig[i]*real_ker[i] for i in range(4)]

print("Frequency-domain product (real part):", real_prod)

# To get back to time domain, we would do an inverse DFT, but we can see that the multiplication in frequency corresponds to convolution in time.
```


## What the code does

First, we define a simple 1D convolution function that loops over the signal, and for each position multiplies the kernel with the overlapping segment and sums them. This is exactly what a convolutional layer does, only in 2D and with many more values. Then we implement a basic discrete Fourier transform. The DFT works by, for each frequency index k, correlating the signal with a cosine (real part) and a negative sine (imaginary part) of that frequency. This gives us the amplitude and phase of that frequency component. We calculate the DFT of both the signal and the kernel (padding the kernel to the same length to make the multiplication valid). Then we multiply the complex numbers in the frequency domain, which is a standard operation. The key point is that the product in frequency corresponds to the convolution in space — this is what the convolution theorem guarantees.


## Common misconceptions

**Spectral bias is a bug that can be fixed** — The network's preference for low frequencies is a flaw that I can eliminate with a better algorithm.

Spectral bias is an inherent feature of gradient descent on these loss landscapes; it arises because high-frequency components have a more ill-conditioned landscape. It's not a bug that can be fully removed, though it can be influenced by techniques like learning rate schedules or changing the architecture (e.g., using attention instead of convolution). In fact: Spectral bias is a property, not a bug. Understanding it helps you predict and manage training dynamics, such as why the model learns shapes before details.

**Fourier analysis is only for audio** — Fourier analysis is a tool for signals like sound; it doesn't apply to image recognition or NLP.

Any function of space or time can be decomposed into frequencies, including pixel intensities across an image or token embeddings across a sequence. Convolutional networks on images naturally operate in the frequency domain via the convolution theorem. In fact: Fourier analysis applies to any continuous or discrete signal, including images and sequences. It's used to analyse and design filters in these domains.

**Learning low frequencies first means the model is dumb** — If the model is learning the 'easy' patterns first, it must be stupid or underfitting.

Low-frequency structure often captures the most generalisable and fundamental patterns, which is beneficial. The model is not being lazy; it's taking the path of least resistance in the optimisation landscape. In fact: Learning low-frequency first is a sign of healthy learning dynamics and often leads to better generalisation, as the model captures the main structure before detailing noise.

**The convolution theorem means the Fourier transform is only for analysing after training** — Fourier analysis is just a diagnostic tool, not something used during the training or architecture of a network.

The theorem can inspire architectures, e.g., FFT-based convolutions which are more efficient for large kernels. It also underlies many network designs like spectral normalisation or frequency-domain feature engineering. In fact: Fourier analysis is used both for post-hoc analysis and for designing efficient algorithms and architectures. Some models explicitly transform to the frequency domain to speed up operations.

**A network that learns high-frequency noise is just 'overfitting' and nothing more** — When my model starts to fit noise, it's just overfitting, and I can fix it by more data or regularisation.

Overfitting is a manifestation of the network's ability to learn high-frequency details, which is a natural part of learning. The issue is that it might start to do so too early or too strongly. Regularisation helps, but understanding the frequency balance can lead to smarter solutions like constraining the network's spectral complexity. In fact: Overfitting to high-frequency noise is a natural result of the network's capacity to represent them. Effective countermeasures often involve controlling the frequency content of the data or the network itself, not just reducing capacity.

**Convolution and Fourier transforms are separate things** — Convolution is what happens in the neural network, and Fourier transforms are a separate mathematical tool used for analysis — they don't interact.

The convolution theorem shows they are mathematically equivalent: applying a filter in space is the same as multiplying in the frequency domain. This is not just a theoretical nicety; it allows for computational shortcuts and deep insights. In fact: Convolution in the spatial domain is equivalent to multiplication in the frequency domain. This theorem is both practically useful and conceptually unifying.

**Fourier analysis can tell me the exact weights my network will learn** — If I analyse the data's frequencies, I can perfectly predict the final trained weights.

The training process involves non-linearities (activation functions, gradient descent paths) that mix frequencies. The spectral bias is a tendency, not a deterministic rule. In fact: Fourier analysis describes general tendencies and helps explain why certain solutions are preferred, but it cannot precisely predict the outcome due to non-linearity.

**All networks have the same spectral bias regardless of architecture** — The bias toward learning low frequencies first is a universal constant; my transformer model will behave the same as a convolutional network.

The spectral bias depends on the architecture, activation function, loss landscape, and even the data distribution. For example, transformers with attention may exhibit different frequency biases than convolutional networks, often leaning to different inductive biases. In fact: While a general spectral bias exists across many models, its detailed behaviour (how strong, how quickly) varies. You should empirically inspect the frequency behaviour of your specific model.


## In short

Fourier analysis reveals how networks process and learn frequencies. Convolution in space equals multiplication in frequency, linking the two views. Networks exhibit a spectral bias, learning low frequencies first and often more robustly, which helps explain generalisation and guides practical training and architecture decisions.


## Where this is used

In large language models (LLMs), which process text as a sequence of tokens, Fourier analysis reveals how the model attends to different frequencies in the token embeddings. The attention mechanism itself has a spectral interpretation: it can be seen as a filtering operation in the frequency domain, where the model learns to attend to low-frequency (global, syntactic) patterns versus high-frequency (local, semantic) ones. Developers of LLMs use Fourier analysis to diagnose training issues: if a model is underperforming, they might look at the frequency spectrum of the model's hidden states to see if it is over-emphasising high-frequency noise (overfitting) or too heavily smoothing (underfitting). This can guide them to adjust the architecture or the training data.

In small language models running on devices like phones, the computational constraints are tight. Fourier analysis helps in model compression: since low-frequency components are often the most important, developers can prune the high-frequency weights, which are often redundant, by applying a Fourier transform to the weight matrices and zeroing out high-frequency coefficients. This reduces the model size with minimal loss of performance. In practice, they might use a library like `torch.fft` to transform the weights, then threshold them, and then transform back.

In retrieval-augmented generation (RAG), embeddings of text are vectors. The similarity between a query and a document is often computed using a dot product. Fourier analysis can be applied to the embedding space to understand what frequencies of meaning are captured. For instance, models often create embeddings that cluster by topic — low-frequency topics are broad, while high-frequency topics are specific details. By applying a Fourier transform to embeddings, developers can filter out high-frequency noise that might not be relevant to the semantic match, improving retrieval accuracy. This is done by transforming the embeddings, keeping only the low-frequency components, and then computing the similarity.

In agentic systems that plan and call tools, the planning process often involves predicting sequential actions. Fourier analysis can help in understanding the 'temporal frequency' of actions: some actions occur frequently, others rarely. The agent's policy network, which decides which action to take, can be analysed in the frequency domain to see if it is too biased toward high-frequency (fast, reactive) actions versus low-frequency (planful) actions. This might lead a developer to adjust the training to balance the two. For example, if the network is too reactive, they might increase the weight of low-frequency components in the loss function.

In everyday applied ML — vision, forecasting, fraud detection, speech — Fourier analysis is pervasive. In vision, as mentioned, convolutional networks can be understood as learning frequency filters. In time-series forecasting, the input sequence has an obvious frequency content (daily cycles, weekly cycles). By taking the Fourier transform of the input, the model can explicitly learn to predict at different frequencies. In fraud detection, the 'signal' is the sequence of user transactions; Fourier features can be engineered to capture periodic fraud patterns. In speech, the audio is naturally analysed in frequencies, with concepts like the spectrogram being a time-varying Fourier transform. Developers of speech models often pre-engineer features using the Fourier transform (e.g., Mel-frequency cepstral coefficients).

When a developer wants to apply these ideas, they typically use a library like `numpy.fft` or `torch.fft`. They might write code like `freq = torch.fft.rfft(signal, dim=-1)` to get the frequency components, then `filtered = freq * mask` to zero out certain frequencies, and then `signal_filtered = torch.fft.irfft(filtered, dim=-1)` to go back. The 'mask' is a learned or hand-crafted binary vector. The common failure mode is that if you filter out too many high frequencies, you lose detail; if you keep too many, you amplify noise. The spectral bias of the network will also cause it to learn low frequencies faster, so a common debugging step is to look at the frequency content of the learned filters and the loss curve.


---

Read this on the web: https://basavapurushottam.com/learn-ai/fourier-analysis-of-networks
