Introduction to Implicit Neural Representations

Welcome to a new module! We've journeyed through the core principles of modern generative models, culminating in our understanding of how Denoising Diffusion Probabilistic Models (DDPMs) can synthesize high-fidelity 2D images. Now, we pivot from generating images to representing and rendering entire 3D worlds. This is the domain of neural rendering, a field that has been revolutionized in recent years.

Our first step is to understand the fundamental data structure at the heart of these new techniques. In this lesson, we will define the concept of an implicit neural representation (INR). You will learn how a simple multilayer perceptron (MLP) can be used in a non-obvious way: not to classify or regress on a dataset, but to become a continuous function that represents a complex signal, like a 3D shape or scene. This concept is the essential prerequisite for dissecting the groundbreaking NeRF paper in our upcoming lessons.

From Discrete Objects to Continuous Functions

In your computer graphics studies, you are already familiar with several ways to represent 3D geometry. The most common are explicit representations:

• Polygon Meshes: A collection of vertices, edges, and faces (usually triangles) that define the boundary surface of an object.
• Voxel Grids: A 3D grid where each cell (voxel) stores a value, such as occupancy (is this cell filled or empty?).
• Point Clouds: A simple set of (x, y, z) coordinates representing points on an object's surface.

These representations are discrete. They approximate a continuous surface with a finite number of elements. This leads to trade-offs: a high-resolution mesh or voxel grid can consume enormous amounts of memory, and point clouds don't inherently define a continuous surface at all.

An alternative is an implicit representation, where a shape is defined by a continuous function. For example, a sphere centered at the origin with radius 1 can be implicitly defined by the function $F(x,y,z)=x^2+y^2+z^2-1$. Any point $(x,y,z)$ for which $F(x,y,z)=0$ lies on the surface. Points where $F<0$ are inside, and points where $F>0$ are outside.

The following video introduces this idea, contrasting these traditional discrete representations with the more powerful continuous, implicit approach.

A particularly useful type of implicit function is the Signed Distance Function (SDF). An SDF for a given shape is a function that, for any point in 3D space, returns the shortest distance to the shape's surface. The sign of the distance is negative if the point is inside the shape and positive if it's outside. The surface itself is therefore the "zero-level set" of the function, where SDF(x, y, z) = 0.

The "Neural" in Implicit Neural Representation

Simple mathematical formulas can only describe simple shapes like spheres and planes. The real world is far more complex. This is where neural networks enter the picture.

An Implicit Neural Representation (INR) uses a neural network, typically a simple MLP, to act as the implicit function. The network learns to approximate a function that maps an input coordinate to a corresponding signal value. The network's weights, $\theta$, become the representation of the signal itself.

For representing a 3D shape, we can train an MLP to approximate its SDF. The network takes a 3D coordinate $(x,y,z)$ as input and outputs a single scalar value: the predicted signed distance at that point.

$${\text{MLP}}_{\theta }:(x,y,z)\to d$$

where $d$ is the signed distance. The network is trained by sampling many points in space and using their known SDF values as ground truth. This approach was popularized by the paper DeepSDF.

Let's watch two short clips that clarify this core idea. The first explains why the "neural" part is necessary for complexity, and the second details how an SDF works.

This is a profound shift in how we use neural networks. Instead of training a network on a large dataset of different shapes to learn a general shape classifier or regressor, we are training (or "overfitting") a network to represent a single, specific instance. The entire, complex geometry of one object is encoded within the weights of one MLP.

Socratic Question: A triangle mesh for a high-resolution object can be megabytes or even gigabytes. An MLP with millions of parameters can also be large. What advantages might representing a shape as a continuous function (the MLP) offer over a discrete collection of triangles, especially regarding resolution, memory, and computing surface properties like normals?

...

The key advantages of an INR lie in its continuous and differentiable nature:

1. Resolution-Independent: The representation is not tied to a grid or a fixed number of polygons. To get a higher-resolution surface, you simply query the function at more points. The memory cost (the size of the network's weights) remains constant.
2. Memory-Efficient: For very complex shapes, the size of the network weights can be significantly smaller than a high-resolution mesh or voxel grid needed to capture the same level of detail.
3. Differentiability: Because the MLP is a differentiable function, we can compute its gradient with respect to the input coordinates $(x,y,z)$. The gradient of an SDF, $\nabla F$, points in the direction of steepest ascent, which is perpendicular to the level sets. At the surface (where $F=0$), this gradient gives us the surface normal for free—a crucial property for lighting calculations in rendering.

From Shape to Scene: The NeRF Representation

Now, let's extend this concept from representing a solid shape to representing a full 3D scene for rendering. This is the core idea behind Neural Radiance Fields (NeRF).

A NeRF doesn't just need to know where a surface is. To render a photorealistic image, it needs to know how light interacts with every point in a volume. The INR for a NeRF is therefore trained to learn a "radiance field," which maps a coordinate to the physical properties needed for rendering:

• Volume Density ($\sigma$): A scalar value representing how opaque that point in space is. A high density means the point is likely part of a solid object, while zero density means it's empty space.
• Color ($R,G,B$): A three-channel vector representing the color of the light emitted from that point in space.

Furthermore, to capture realistic effects like reflections and specular highlights, the appearance of a point can change depending on the angle from which it is viewed. Therefore, the NeRF MLP takes not just a 3D position but also a 2D viewing direction as input.

The complete function learned by a NeRF is:

$$F_{\theta }:(\mathbf{x},\mathbf{d})\to (\mathbf{c},\sigma )$$

where:

• $\theta$ are the network weights.
• $\mathbf{x}=(x,y,z)$ is the 3D location.
• $\mathbf{d}=(\theta ,\varphi )$ is the 2D viewing direction.
• $\mathbf{c}=(R,G,B)$ is the emitted color.
• $\sigma$ is the volume density.

The following diagram from the original NeRF paper perfectly illustrates this process.

The next video provides a concise explanation of this specific 5D function.

This concept is formalized very well in the following reading.

Conclusion

In this lesson, we have defined the fundamental concept that underpins modern neural rendering. You now understand that an implicit neural representation is not just another type of neural network architecture, but a paradigm for using a network as a continuous, differentiable, and memory-efficient representation of a signal.

Key Takeaways:

• Traditional 3D representations like meshes and voxels are explicit and discrete.
• An implicit representation defines a shape or scene using a continuous function $F(\text{coordinate})=0$. A Signed Distance Function (SDF) is a powerful example.
• An Implicit Neural Representation (INR) uses an MLP to learn this continuous function, encoding a complex signal (like a 3D shape) into the network's weights.
• The specific INR used by NeRF learns a 5D function that maps a 3D position $(x,y,z)$ and a 2D viewing direction $(\theta ,\varphi )$ to a color $(R,G,B)$ and a volume density $(\sigma )$.

Preview of the Next Lesson:

A standard MLP has a significant weakness: a "spectral bias" that makes it very good at learning low-frequency signals (smooth shapes) but very poor at learning the high-frequency details (sharp edges, fine textures) needed for photorealism. In our next lesson, we will explore this problem and introduce the simple yet powerful technique of positional encoding, which was the key that unlocked the high-fidelity results of NeRF.