3D Generation with Diffusion Models

1. From Implicit to Explicit Generation

So far, our process has been:
Text Prompt → 2D Diffusion Model (Guidance) → NeRF (Implicit Rep)

This is an indirect method that cleverly repurposes 2D models. Today, we explore a more direct approach:
Text Prompt → 3D Diffusion Model → Point Cloud / Voxels (Explicit Rep)

The main advantage of explicit representations is that they define the geometry of a shape directly.

• Point Clouds: A set of vertices in 3D space, often with color attributes.
• Voxel Grids: A 3D grid where each cell (voxel) can be occupied or empty, similar to pixels in a 2D image.

These formats are standard in 3D scanning, robotics, and computer graphics, making them easier to edit, convert to meshes, and import into other applications.

2. Diffusion on Point Clouds

How can we apply a diffusion process to a point cloud? A point cloud is an unordered set of 3D coordinates. We can treat a cloud of $N$ points as a single tensor of shape $(N,3)$ and apply the diffusion process you're already familiar with.

The forward process involves gradually adding Gaussian noise to the coordinates of each point until the structured shape dissolves into a random cloud. The reverse process involves training a neural network to denoise this cloud, step-by-step, back into a coherent object.

The main architectural challenge is that the denoising network must be permutation-invariant. The order of points in the input tensor doesn't change the object they represent, so the network's output shouldn't change if the points are shuffled. This is a classic problem in 3D deep learning, and networks like PointNet or specialized convolutions are often used.

Point-Voxel Diffusion (PVD)

One of the key challenges is choosing a representation that is amenable to the diffusion process. Pure point clouds require permutation-invariant networks, while pure voxel grids can be sparse and memory-intensive. The "Point-Voxel Diffusion" (PVD) paper proposes a hybrid approach.

As you read, you'll see that PVD models the problem as denoising a tensor of point coordinates $x\in {\mathbb{R}}^{N\times 3}$. The loss function is the familiar L2 loss between the true and predicted noise:

The key difference from 2D diffusion is that $x_t$ is a noised point cloud and ${ϵ}_{\theta }$ is a network designed for 3D data.

Latent Point Diffusion

Another powerful strategy, similar to latent diffusion for images, is to first encode the point cloud into a more compact latent space and then run the diffusion process there. This is the approach taken by models like LION.

This latent-space approach is computationally efficient and allows the model to focus on the high-level structure of the object during the diffusion process.

Practical Example: OpenAI's Point-E

Point-E is a well-known model that generates point clouds from text prompts. It uses a two-stage diffusion process:

1. A text-to-point-cloud model generates a low-resolution (1,024 points) point cloud.
2. An upsampling diffusion model takes this low-resolution cloud and generates a high-resolution (4,096 points) version.

This cascaded approach allows for efficient generation of detailed shapes. The following video demonstrates how you can use Point-E.

This practical workflow—generate points, then convert to mesh—is a common pattern for models that output explicit point cloud representations.

3. Diffusion on Voxel-Based Representations

Voxel grids are another popular explicit representation. However, a naive dense voxel grid suffers from the curse of dimensionality: a $512\times 512\times 512$ grid contains over 134 million voxels, making it computationally infeasible for diffusion models.

The solution is to use sparse and/or hierarchical representations that only store information where the geometry actually exists.

Hierarchical Latent Diffusion with XCube

XCube is a model that uses a sparse voxel hierarchy. It's a great example of a modern, efficient approach.

The core idea is to generate the shape coarse-to-fine. A diffusion model first generates a low-resolution base structure, and subsequent models add detail at higher resolutions, conditioned on the previous levels. This is much more efficient than trying to generate the entire high-resolution grid at once.

Octree-based Diffusion with OctFusion

Octrees are a classic data structure for efficiently representing sparse 3D data. They are trees where each node has up to eight children, corresponding to the eight octants of a 3D space. A region is recursively subdivided until a desired level of detail is reached or a voxel is empty.

Recent models like OctFusion have combined octrees with latent diffusion to achieve state-of-the-art results.

The OctFusion architecture is a prime example of modern trends in generative 3D modeling:

1. Find an efficient representation: Instead of dense grids, use a smart, sparse structure like an octree.
2. Compress it: Use a VAE to map this structure into a compact latent space.
3. Diffuse in latent space: Train a diffusion model to generate novel latents. This is much cheaper than diffusing in the raw data space.
4. Decode the result: Use the VAE's decoder to turn the generated latent back into a high-resolution 3D shape.

4. Summary: Implicit vs. Explicit 3D Generation

Let's summarize the trade-offs between the SDS-based implicit methods and the direct diffusion-based explicit methods we've covered today.

Conclusion

In this lesson, you've learned how diffusion models can be applied to generate explicit 3D representations. This is a powerful alternative to the SDS-based methods we studied previously.

Key Takeaways:

• Diffusion models can be trained directly on 3D data like point clouds and voxel grids.
• For point clouds, the denoising network must be permutation-invariant.
• For voxel grids, sparse and hierarchical representations (like octrees) are essential to overcome the curse of dimensionality.
• A common and effective architectural pattern is to use a VAE to learn a compact latent space for the 3D data, and then apply the diffusion process within that latent space.
• Explicit generation is typically much faster than implicit generation via SDS and produces editable assets, but it requires large 3D training datasets.

Next Lesson Preview:

We've now seen two distinct paradigms: text-to-3D using 2D priors (SDS) and direct generation from 3D datasets. The next module will explore techniques that bridge these worlds. In the next lesson, we will focus on setting up a framework for single-view 3D reconstruction, where the goal is to optimize a NeRF to represent an object shown in a single input image. This is a fundamental task that moves us closer to creating 3D content from specific visual inputs.