Debugging 3D Artifacts

1. The Anatomy of a Failed 3D Generation

When you optimize a NeRF with SDS, you're relying entirely on the "opinions" of a pre-trained 2D diffusion model. If that 2D model has biases or fails to understand your prompt perfectly from every angle, the resulting 3D object will inherit those flaws.

The Janus Problem

The most notorious artifact is the Janus problem, named after the two-faced Roman god. This is when a generated object has multiple "fronts" instead of a coherent front, back, and sides. For example, a model of a person might have a face on both the front and back of its head.

The cause lies in the 2D diffusion model's training data. Models like Stable Diffusion have seen millions of images of people, and the vast majority are portraits or photos showing their face. The model develops an extremely strong prior that "person" implies "face."

In the last lesson, we used view-dependent prompts like "a person, back view". But often, the signal from "person" is so strong that it overpowers the "back view" qualifier. When your training loop renders the NeRF from the back, the 2D diffusion model still criticizes it for not having a face, so the optimizer dutifully adds one.

Geometric Inconsistencies

Other common issues include:

• "Floaters": Disconnected pieces of geometry that hover around the main object. This often happens when the diffusion model produces an image with a complex background, and SDS tries to interpret that background as part of the 3D object.
• Flat Geometry / "Billboarding": The object looks good from the front, but when viewed from the side, it's revealed to be nearly flat. This occurs when the model finds a "lazy" solution that satisfies the 2D views without forming a true volumetric shape.

2. A Principled Solution: Perpendicular Negative Prompts (Perp-Neg)

To combat the Janus problem, we need a stronger way to control the diffusion model's guidance. Simply adding "back view" to the prompt isn't enough. The key insight is to simultaneously tell the model what we want (the positive prompt) and what we don't want (the negative prompt).

This is the idea behind the Perp-Neg algorithm.

Deconstructing the Perp-Neg Solution

As you read, you'll see the paper proposes a two-part solution: a clever prompting strategy and a modification to the guidance calculation.

1. View-Specific Negative Prompting

The core idea is to use other views as negative prompts. For instance, to generate the back of an object, you explicitly tell the model not to generate the front or side. Based on the paper, the strategy is:

This gives the model unambiguous instructions. You aren't just hoping it understands "back view"; you're actively penalizing it for generating a front or side view instead.

2. Perpendicular Guidance

A naive implementation of negative prompts can fail if the positive and negative prompts have significant overlap (e.g., + "a corgi, back view" and - "a corgi, front view" both share "a corgi"). This can cause the optimizer to destroy the object entirely.

Perp-Neg solves this by modifying the score calculation. In standard classifier-free guidance, the final score estimate is a weighted combination of the conditional and unconditional scores. Perp-Neg adds a third component: the negative prompt's score.

Crucially, it projects the negative score vector to be perpendicular to the positive score vector before subtracting it. In your ML background, you can think of this as performing a Gram-Schmidt-like step. This process removes the components of the negative guidance that align with the positive guidance (e.g., the shared "corgi" concept) while preserving the parts that are orthogonal (the conflicting "front view" vs. "back view" information).

The final noise prediction in the SDS loss (Equation 12 from the paper) becomes:

This new noise estimate, ${\widehat{\mathbf{ϵ}}}_{\mathbf{\theta }}^{\text{PN}}$, is then plugged directly into the SDS loss function you implemented in the previous lesson, replacing the simpler CFG term.

The stable-dreamfusion repository, a practical implementation of these ideas, has integrated this technique.

3. Practical Debugging and Alternative Strategies

While Perp-Neg is a powerful solution, debugging 3D generation is also an empirical process.

Visualizing Guidance

One of the most effective debugging tools is to visualize the 2D diffusion model's output at each step. The stable-dreamfusion repository provides a command-line flag for this.

By saving the images that the SDS loss is calculated on, you can directly see what the diffusion model is "thinking." If you are rendering from a back-facing camera but the saved guidance image shows a face, you have visual confirmation of the Janus problem. This allows you to inspect whether your prompting strategies (like Perp-Neg) are having the desired effect.

Alternative and Complementary Methods

The Janus problem is an active area of research, and Perp-Neg is not the only approach. Other methods focus on "debiasing" the diffusion model's score estimates or the prompts themselves.

This image illustrates that you can either intervene at the prompt level (like Perp-Neg does) or at the score level. Score debiasing acts as a filter on the diffusion model's output, attempting to clean up the guidance signal before it's used to update the NeRF. This shows that improving 3D generation is a multi-faceted problem, and robust solutions often combine several techniques.

Conclusion

You've now moved beyond the basic text-to-3D pipeline and into the critical domain of debugging and refinement. You have learned that many common artifacts are not bugs in your code, but emergent properties of using a biased 2D model as a guide.

Key Takeaways:

• The Janus (multi-head) problem is a primary failure mode caused by the canonical view bias in pre-trained 2D diffusion models.
• The Perp-Neg algorithm provides a robust solution by using view-specific negative prompts and projecting their guidance to be perpendicular to the positive prompt's guidance.
• This is implemented by modifying the noise prediction term within the SDS loss function, making it a targeted change to the training loop.
• Practical debugging involves visualizing intermediate guidance images to diagnose if the 2D model is correctly interpreting the view-dependent prompts.
• This is an active field of research with alternative solutions like score and prompt debiasing.

Next Lesson Preview:

So far, we have focused on generating 3D objects as implicit representations using NeRFs. While powerful for rendering, NeRFs can be difficult to edit or use in traditional 3D software. In our next lesson, we will shift focus and learn how to apply diffusion models to generate explicit 3D representations like point clouds or voxels, opening up new possibilities for asset creation and manipulation.