3D Representations for Diffusion Models

1. A Tale of Two Geometries: Explicit vs. Implicit

At a high level, 3D representations fall into two categories:

• Explicit Representations: These directly define the object's geometry. They consist of a list of geometric primitives like points, cubes, or polygons that make up the shape. Voxels, point clouds, and meshes fall into this category.
• Implicit Representations: These define an object's geometry indirectly using a function. Instead of storing the surface, you learn a function $f(\mathbf{p})$ that tells you something about any given point $\mathbf{p}=(x,y,z)$ in space (e.g., is it inside or outside the object?). Neural fields are the primary example here.

Let's begin our comparison with the explicit types.

2. Explicit Representations: Building Blocks of 3D

The most intuitive way to describe a 3D shape is by listing its constituent parts. This is the core idea behind explicit representations. To get a feel for what these look like, consider the famous Stanford Bunny rendered in three different explicit formats.

Now, let's break down each one. The following video provides a great overview of these representations and the deep learning challenges they introduce.

Building on the concepts from the video, let's formalize our comparison. For a more structured summary of these representations, the following survey paper is an excellent resource.

Now let's synthesize this information, focusing on the implications for diffusion models.

2.1 Voxel Grids

A voxel grid is the most direct 3D analogue to the 2D pixel grid used in images. 3D space is discretized into a uniform grid, and each cell (voxel) stores a value, typically representing occupancy (is this part of the object?) or color.

• Pros: The regular, grid-like structure is a major advantage. It allows the direct application of 3D Convolutional Neural Networks (3D CNNs), making the transition from 2D image models conceptually straightforward.
• Cons: The memory and computational costs are prohibitive. The number of voxels grows cubically with resolution ($N^3$). A $512\times 512$ image is manageable, but a $512\times 512\times 512$ voxel grid is enormous. This severely limits the resolution and detail of generated objects, often resulting in "blocky" artifacts.
• Diffusion Model Application: A diffusion model for voxels would be a U-Net with 3D convolutional layers. The process is analogous to image diffusion: start with a 3D grid of Gaussian noise and iteratively denoise it to produce a final voxel grid. The main challenge is managing the immense memory footprint during training and inference. Techniques like sparse voxel octrees, mentioned in the video, are often used to mitigate this.

2.2 Point Clouds

A point cloud is perhaps the simplest and most raw 3D representation: an unordered set of points $\{{\mathbf{p}}_i{\}}_{i=1}^N$, where each ${\mathbf{p}}_i=(x_i,y_i,z_i)$. This is often the direct output of 3D scanners like LiDAR.

• Pros: They are memory-efficient and directly represent surface geometry without imposing any topological constraints (e.g., they can represent any shape, with or without holes, without issue).
• Cons: Their primary challenge is their lack of structure. They are unordered sets, not grids. A standard CNN cannot process them because there's no fixed neighborhood or order. This also means there's no surface connectivity information, making high-quality rendering difficult without post-processing.
• Diffusion Model Application: To handle the unordered nature of point clouds, a diffusion model must use a permutation-invariant backbone, such as PointNet or a Graph Neural Network (GNN). The diffusion process learns to denoise a set of points, starting from a random Gaussian cloud and progressively moving the points onto the surface of the target shape.

To understand how this works in practice, let's focus on the deep learning architectures for point clouds and their use in generative models.

2.3 Meshes

Meshes are the standard for 3D computer graphics and modeling. They consist of a set of vertices (points in space) connected by edges to form faces (typically triangles or quads).

• Pros: They are memory-efficient and explicitly define a continuous surface with topological information. This makes them ideal for high-quality, fast rendering and physical simulations.
• Cons: Their irregular, graph-like connectivity makes them challenging for standard deep learning models. More importantly, the mesh topology (the connectivity of vertices) is typically fixed. This is a major limitation for generative models, as generating a shape with a different number of holes than the template mesh is non-trivial.
• Diffusion Model Application: This is an active area of research. One approach is to apply diffusion to the vertex positions of a template mesh. The model learns to deform the template into the target shape. However, this cannot change the topology. More advanced methods attempt to generate mesh elements (vertices and faces) autoregressively or use Graph Neural Networks (GNNs) as the backbone, but this remains a complex problem.

3. Implicit Representations: Neural Fields

Instead of listing geometric primitives, implicit representations define a shape with a continuous function $f(\mathbf{p})$ that can be queried at any 3D coordinate $\mathbf{p}$. This function is typically represented by a Multi-Layer Perceptron (MLP). This family of representations is often called neural fields.

The two most common types are:

• Occupancy Fields: The function outputs a probability of occupancy, $f(\mathbf{p})\to [0,1]$. The surface is the decision boundary where $f(\mathbf{p})=0.5$.
• Signed Distance Fields (SDFs): The function outputs the signed distance to the nearest surface, $f(\mathbf{p})\to d$. The sign indicates whether the point is inside ($<0$) or outside ($>0$) the object. The surface is the zero-level set, where $f(\mathbf{p})=0$.

• Pros: Because they are continuous, they can represent shapes at arbitrary resolution. They are extremely memory-efficient, as the entire shape is encoded in the weights of a small MLP. They are also topologically flexible, able to represent any shape without constraints.
• Cons: They are not explicit. To visualize a neural field, you must perform a post-processing step to extract an explicit representation (like a mesh), often using an algorithm like Marching Cubes. This can be computationally intensive.
• Diffusion Model Application: This is where things get interesting and powerful. A diffusion model typically does not generate the geometry directly. Instead, it operates in a latent space. For example, a model could be trained to generate a latent vector $\mathbf{z}$ that conditions the MLP: $f(\mathbf{p},\mathbf{z})\to d$. By diffusing and sampling in the space of $\mathbf{z}$, you can generate a variety of shapes. In upcoming lessons, we'll see an even more powerful technique (Score Distillation Sampling) where a pre-trained 2D diffusion model guides the optimization of a 3D neural field.

4. Head-to-Head Comparison

To consolidate our understanding, this table summarizes the key trade-offs.

Conclusion

Today we've laid the essential groundwork for our exploration of 3D diffusion. We've seen that choosing a 3D representation is a critical design decision with significant trade-offs. There is no single "best" format; the optimal choice depends heavily on the specific application, desired output quality, and available computational resources.

Key Takeaways:

• Explicit representations (voxels, point clouds, meshes) define geometry directly, while implicit representations (neural fields) use a function.
• Voxels are simple for CNNs but suffer from cubic memory scaling.
• Point Clouds are simple and flexible but require specialized, permutation-invariant architectures like PointNet.
• Meshes are standard for graphics but their fixed topology is a major hurdle for generative models.
• Neural Fields (SDFs, Occupancy Fields) are memory-efficient, continuous, and topologically flexible, making them a powerful choice for modern generative models, though they require an extraction step for visualization.
• The choice of representation fundamentally dictates the architecture of the diffusion model, from 3D U-Nets for voxels to permutation-invariant networks for point clouds to operating in a latent space for neural fields.

Next Lesson Preview:

We've introduced the concept of neural fields as a powerful way to represent geometry. In our next lesson, we will dive deep into the most influential neural field architecture: Neural Radiance Fields (NeRFs). We'll move beyond just shape and learn how NeRFs can represent a full 3D scene with photorealistic appearance. Understanding the NeRF rendering process will be a cornerstone for the advanced text-to-3D generation techniques we'll tackle later in the course.