Diffusion Priors for Single-View NeRF

1. The Core Idea: Distilling a 2D Prior into a 3D Representation

Imagine you have a NeRF that is randomly initialized. If you render an image from a random viewpoint, it will look like noise. How can we tell the NeRF how to improve? We need a "teacher" or a "critic" that knows what plausible objects look like. This is where a pre-trained text-to-image diffusion model comes in.

The central mechanism we'll use is called Score Distillation Sampling (SDS), first introduced in the DreamFusion paper. The key insight is that we can use the gradients from a frozen, pre-trained 2D diffusion model to update the parameters of our 3D representation (the NeRF).

To get a solid grasp of this concept, let's watch a couple of short videos that explain it from different angles.

These videos explain SDS in the context of text-to-3D. Let's formalize the process and visualize it before adapting it to single-view reconstruction.

The iterative SDS update loop for a single optimization step looks like this:

1. Render: Generate a 2D image $x=g(\theta )$ by rendering your NeRF (with parameters $\theta$) from a random camera pose.
2. Noise: Add random Gaussian noise $ϵ$ to the image to get a noisy version $x_t$ at a random timestep $t$.
3. Predict Noise: Use the frozen U-Net of a pre-trained diffusion model, ${ϵ}_{\varphi }$, to predict the added noise. This prediction is conditioned on a text prompt $y$: ${ϵ}_{\text{pred}}={ϵ}_{\varphi }(x_t,y,t)$.
4. Compute Gradient: The core of SDS is to treat the difference between the predicted noise and the actual noise as a loss. The gradient of this loss with respect to the NeRF parameters $\theta$ is used for the update: $${\nabla }_{\theta }{\mathcal{L}}_{\text{SDS}}\propto ({ϵ}_{\text{pred}}-ϵ)\frac{\partial x}{\partial \theta }$$ This gradient essentially tells the NeRF how to change its parameters $\theta$ so that its render $x$ becomes an image that the diffusion model considers more plausible for the given text prompt $y$.

2. Adapting SDS for Single-View Reconstruction: The RealFusion Method

The process described above generates a 3D object from a text prompt. Our goal, however, is to reconstruct a specific object from a single image. We need to make two crucial adaptations:

1. Grounding in the Input Image: We must ensure the final 3D model looks exactly like the input image from the known camera view.
2. Conditioning the Prior: The diffusion model shouldn't just dream up any object matching a generic prompt; it must dream up novel views of the specific object in our input image.

A great case study for this is the RealFusion model. Its methodology combines a traditional reconstruction loss with the SDS prior loss.

As you saw, the optimization process involves a hybrid objective:

• Reconstruction Loss: For the known input camera pose, render an image from the NeRF and compute a simple photometric loss (e.g., L2) against the input image. This forces the NeRF to be faithful to the provided view. \mathcal{L}_\text{recon} = || g(\theta_\text{input_pose}) - x_\text{input} ||^2
• Prior Loss: For randomly sampled novel views, render images from the NeRF and apply the SDS loss ${\mathcal{L}}_{\text{SDS}}$. This uses the diffusion prior to fill in the occluded and unseen parts of the object.

The total loss is a weighted sum of these two components.

Conditioning the Diffusion Prior with Textual Inversion

How do we guide the SDS loss? A simple prompt like "a photo of a teapot" is not enough to reconstruct the specific blue-and-white patterned teapot from our input image. We need to teach the diffusion model about this specific instance.

RealFusion solves this by using single-image textual inversion.

Once this <object> token is learned, it can be used in the prompts for the SDS loss, enabling highly specific guidance like "a photo of a <object>, back view". This brings us to another critical detail for getting good geometry.

3. Implementation Details and Practical Gotchas

As an engineer, you know that the high-level idea is only half the story. The details of the implementation are what make it work. Let's dig into the supplementary material for RealFusion, which is a treasure trove of practical insights.

Let's synthesize those practical details into a coherent pipeline:

1. Preprocessing (Textual Inversion): Given a single input image, run single-image textual inversion for ~3000 steps with heavy augmentation to get a custom embedding <object> that represents your specific object.
2. Main Optimization Loop (e.g., 5000 steps): In each step:
   • Reconstruction Step:
     • Render the NeRF from the known input camera pose.
     • Compute an L2 loss against the ground truth input image.
   • Prior Step:
     • Sample a random camera pose (e.g., random azimuth, elevation).
     • Generate a view-dependent prompt (e.g., "a photo of <object>, side view").
     • Render a low-resolution (96x96) image from the NeRF.
     • Upsample it to 512x512 and encode it into a latent with the VAE.
     • Perform the SDS update using this latent, the prompt, and a high CFG scale.
   • Update: Apply the gradients from both losses to the NeRF parameters.

Conclusion

In this lesson, we have connected the worlds of NeRFs and diffusion models to tackle the challenging task of single-view 3D reconstruction. You've learned how a powerful, pre-trained 2D generative model can act as a prior to guide the optimization of a 3D scene.

Key Takeaways:

• Score Distillation Sampling (SDS) is the core mechanism that allows a frozen 2D diffusion model to provide optimization gradients for a 3D representation like a NeRF.
• For single-view reconstruction, the optimization objective is a hybrid, combining a direct reconstruction loss for the input view with a SDS prior loss for novel views.
• To make the prior specific to the input object, single-image textual inversion is used to create a custom token embedding for the diffusion model's text encoder.
• Practical success depends on key details like using view-dependent prompts to prevent geometric failures (like the Janus problem) and tuning parameters like a high CFG scale.

Next Lesson Preview:

We've focused on reconstructing a single, isolated object. What if our input is not a clean image of one object, but a partial scan of a larger environment, like a single photo of a room? In the next lesson, we will explore how these same diffusion prior techniques can be adapted for 3D scene completion, filling in missing geometry and texture in large-scale scenes from partial data.