Understanding Spectral Bias in MLPs

In our previous lesson, we established the paradigm of an Implicit Neural Representation (INR), where a simple MLP is trained to represent a continuous signal, such as the color and density of a 3D scene. This sets the stage for understanding Neural Radiance Fields (NeRF). However, if you were to implement the NeRF MLP as described—a standard network taking 5D coordinates as input—you would encounter a fundamental and disappointing limitation. The resulting renderings would be blurry and lack the crisp detail that made NeRF famous.

This lesson addresses the "why" behind that failure. We will dissect the concept of spectral bias, a critical shortcoming of standard MLPs that prevents them from representing high-frequency details. Understanding this problem is the first step in appreciating the simple yet profound architectural innovation that allows NeRF to capture photorealistic scenes.

The Problem: A Failure to Represent Detail

Let's begin with a simple but illustrative task: training an MLP to represent a single 2D image. In this setup, the network's input is a 2D pixel coordinate $(x,y)$, and its output is the corresponding $(R,G,B)$ color. This is a classic INR problem. You are essentially asking the network to "overfit" to one image, memorizing the color at every coordinate. Given that a modern MLP can have millions of parameters—far more than the number of pixels in the image—this should be an easy task.

Surprisingly, it is not. A standard MLP struggles mightily. It learns the general, low-frequency information (the overall shapes and smooth color gradients) but fails to reproduce the high-frequency details (sharp edges, textures, and fine patterns).

The following video segment by Jon Barron, one of the authors of NeRF, demonstrates this failure and introduces the solution that we will analyze.

This phenomenon is not unique to images. Any signal with intricate, fine-grained structure will expose this weakness. An excellent example is trying to teach a network to render the Mandelbrot set, a fractal whose boundary is defined by infinite high-frequency detail.

This brings us to a critical question. An MLP is a universal function approximator. In theory, it can represent any continuous function. Yet in practice, it clearly exhibits a strong preference.

Socratic Question: A standard MLP is composed of layers that perform affine transformations (matrix multiplication + bias) followed by a simple non-linearity (like ReLU). Why would a function built this way be inherently better at learning smooth, low-frequency patterns and struggle to generate sharp, high-frequency oscillations?

...

Your intuition might point to the nature of these operations. Matrix multiplications are linear transformations, which are inherently smooth. While the ReLU activation adds non-linearity, it does so in a piecewise-linear fashion. It takes a significant number of these simple "hinges" arranged in a complex configuration to approximate a rapidly oscillating function. The optimization process, like gradient descent, finds it much "easier" to first adjust the weights to fit the coarse, low-frequency components of the target function, effectively getting stuck in a state where it cannot efficiently learn the high-frequency residuals.

The Theory: Spectral Bias and the Neural Tangent Kernel

This observed behavior is formally known as spectral bias: deep neural networks trained with gradient descent exhibit a tendency to learn lower-frequency functions before higher-frequency functions. To understand the mathematical roots of this, we turn to the theory of the Neural Tangent Kernel (NTK).

The NTK is a powerful analytical tool. It shows that an infinitely wide neural network trained with gradient descent is equivalent to a classical method called kernel regression. The "kernel" is a function $k({\mathbf{v}}_1,{\mathbf{v}}_2)$ that measures the similarity between two input points. The specific architecture of the neural network implicitly defines this kernel.

For our purposes, there are two key findings from the NTK literature:

1. MLPs use a Dot-Product Kernel: The kernel implicitly defined by a standard MLP with ReLU activations is a dot-product kernel. This means its output depends on the dot product of the input vectors, $k({\mathbf{v}}_1,{\mathbf{v}}_2)=f({\mathbf{v}}_1\cdot {\mathbf{v}}_2)$.
2. Convergence Rate Depends on the Kernel's Spectrum: The speed at which the network learns different components of the target function depends on the eigenvalues of this kernel. Functions corresponding to large eigenvalues are learned quickly, while functions corresponding to small eigenvalues are learned very slowly.

For the dot-product kernel of an MLP, the eigenvalues associated with high-frequency functions are vanishingly small. This means the convergence rate for these high-frequency components is so slow that the network effectively never learns them in a practical amount of training time.

Let's ground this theory in the following reading from the seminal paper on this topic.

This explains the phenomenon we observed. The MLP is not incapable of representing high frequencies, but the dynamics of gradient-based training make it incredibly difficult to learn them. The network prioritizes fitting the "easy" low-frequency components and makes infinitesimal progress on the "hard" high-frequency details.

The following video segment provides an excellent conceptual explanation, contrasting the MLP's dot-product kernel with the type of kernel one would actually want for an interpolation task.

The image below summarizes the practical effect of this bias. On the top row, a standard MLP tries to fit a complex 1D signal. It only captures the low-frequency trend. On the bottom row, we see the training and test error over time. Notice how the training error for the standard MLP (labeled "No mapping") plateaus quickly, unable to reduce the error further by fitting the high-frequency wiggles. The plots on the far right show the error broken down by frequency components, confirming that the high-frequency error remains stubbornly high.

Conclusion

In this lesson, we have formally identified a critical flaw in using standard MLPs for implicit neural representations. What appears to be a simple memorization task is thwarted by the network's inherent "spectral bias."

Key Takeaways:

• Spectral Bias: Standard MLPs trained with gradient descent learn low-frequency functions much more quickly and easily than high-frequency functions.
• Practical Effect: When used for INRs, this bias results in overly smooth or blurry representations that fail to capture fine details, sharp edges, and complex textures.
• Theoretical Cause: NTK theory reveals that this bias stems from the network's implicit kernel. A standard MLP behaves like a kernel machine with a dot-product kernel whose eigenvalues decay rapidly for high frequencies, leading to impractically slow convergence for those components.

This now frames the key architectural challenge for NeRF: how can we modify our simple MLP to overcome this spectral bias and empower it to learn the high-frequency functions needed for photorealism?

Preview of the Next Lesson:

We will dive deep into the solution: positional encoding, also known as Fourier features. We will explore the mathematics of how this simple input mapping technique effectively transforms the network's implicit kernel, giving it a "tunable bandwidth" that allows it to conquer spectral bias. We will then implement this crucial component from scratch.