Camera-Conditioned 3D Diffusion for Novel View Synthesis

1. The Need for 3D-Aware Priors

As we saw when discussing Score Distillation Sampling (SDS), using a standard 2D text-to-image model as a prior for 3D generation can lead to artifacts. The infamous "Janus problem" (generating objects with two faces) arises because the 2D model lacks an inherent understanding of 3D geometry. It knows what a face looks like, but not that a head should only have one face on the front.

The solution is to explicitly teach the diffusion model about 3D structure. This is done by fine-tuning it on a dataset of 3D objects, where it learns to associate different views with their corresponding camera poses.

As the video explains, by training a model to take a single image and generate a new view based on a relative camera pose, we imbue the model with a "3D prior." This "view-conditional" model is the core component we'll focus on integrating today.

2. Case Study 1: SV3D - Repurposing Video Models for 3D

A highly effective and clever approach to generating multiple consistent views is to treat them as frames in a video. The Stable Video 3D (SV3D) model does exactly this, repurposing a pre-trained video diffusion model (Stable Video Diffusion) for novel view synthesis.

The core idea is to leverage the mechanisms that a video model uses to ensure temporal consistency (e.g., temporal attention, 3D convolutions) to enforce spatial consistency across different camera viewpoints.

So, how is the camera conditioning integrated? This is where we see a different architectural choice compared to Zero-1-to-3.

Let's break down the integration point described in the paper:

1. Pose Representation: The camera pose for each view is simplified to two angles: elevation (e) and azimuth (a).
2. Sinusoidal Embedding: Just like the diffusion timestep t, these angles e and a are converted into high-dimensional vector representations using sinusoidal embeddings. This is the same technique used in Transformers for positional encoding.
3. Embedding Combination: The pose embeddings are concatenated, passed through a linear layer, and then added to the timestep embedding.
4. Injection: This combined embedding (timestep + pose) is then projected and added to the feature maps within each residual block of the U-Net.

This diagram from the SV3D paper shows the integration point. The camera angles (e, a) are embedded and added to the timestep (t) embedding. This combined conditioning is then fed into the convolutional blocks of the U-Net.

This is a crucial architectural pattern. While Zero-1-to-3 injected pose information into the cross-attention mechanism, SV3D injects it into the time and conditioning embedding that modulates the U-Net's residual blocks. Both are valid ways to provide conditioning; this method is very common for non-textual continuous variables like time or, in this case, pose.

3. Case Study 2: SEVA - A Generalist Architecture

Another state-of-the-art model, STABLE VIRTUAL CAMERA (SEVA), takes a slightly different approach. It fine-tunes a powerful image diffusion model (Stable Diffusion 2.1) and augments its architecture to handle multiple views. It demonstrates another common integration pattern.

SEVA presents a third integration strategy:

1. Pose Representation: It uses Plücker embeddings, a 6D representation of 3D lines that is geometrically rich and unambiguous for representing camera rays.
2. Injection via Concatenation: For each view, the camera's Plücker embedding is concatenated to the VAE latent along the channel dimension. This is similar to the channel-wise concatenation we saw in Zero-1-to-3, but here it's done for camera pose.
3. Injection via Modulation: The pose information is also used to modulate the U-Net's layers via adaptive layer normalization (AdaLN). In this technique, the conditioning vector (the pose embedding) is used to predict the scale and bias parameters (gamma and beta) of normalization layers within the network. This allows the pose to have a deep, structural influence on the generation process.

The key takeaway here is that there is no single "correct" way to integrate camera conditioning. The best method depends on the base model architecture and the specific task. As an engineer, knowing these different patterns—cross-attention, additive time embedding, and AdaLN modulation—gives you a versatile toolkit for building your own models.

4. The Integrated Training Pipeline

Now, let's tie these concepts together into a practical training loop for a view-conditional model.

Objective: Fine-tune a pre-trained diffusion model to generate a target view I_tgt given a reference view I_ref and the relative camera pose P_rel between them.

1. Data Preparation:

• You need a large-scale dataset of 3D objects, like Objaverse.
• For each training step, you select a 3D object.
• You define two camera poses, P_ref and P_tgt, and render the corresponding images, I_ref and I_tgt.
• You calculate the relative pose P_rel from P_ref to P_tgt. Your training sample is the triplet: (I_ref, I_tgt, P_rel).
• For multi-view models like SV3D, you would render a whole sequence of target images along a camera trajectory (an "orbit").

2. Model Input and Conditioning:

• Reference Image Conditioning: I_ref is encoded into a latent by the VAE and often also by a CLIP image encoder. This provides the semantic and appearance information. This can be fed via cross-attention or channel-concatenation.
• Camera Pose Conditioning: P_rel is encoded using one of the methods we discussed (e.g., sinusoidal embedding of angles). This provides the geometric guidance.
• Ground Truth: I_tgt is the image the model must learn to generate. It's encoded into a latent z_tgt, and Gaussian noise ε is added to create the noisy latent z_t.

3. The Training Step (High-Level Pseudocode):

# 1. Get a data sample
ref_image, target_image, relative_pose = dataset.get_sample()

# 2. Encode inputs
ref_latent = vae.encode(ref_image)
target_latent = vae.encode(target_image)
pose_embedding = encode_pose(relative_pose) # e.g., sinusoidal embedding

# 3. Prepare for diffusion
noise = torch.randn_like(target_latent)
timestep = torch.randint(0, num_timesteps, (1,))
noisy_target_latent = scheduler.add_noise(target_latent, noise, timestep)
time_embedding = get_time_embedding(timestep)

# 4. Integrate conditioning (SV3D example)
conditioning_embedding = time_embedding + pose_embedding

# 5. Predict noise
predicted_noise = unet(
    noisy_target_latent,
    cond_embedding=conditioning_embedding,
    cross_attention_context=get_clip_embedding(ref_image) # Example context
).sample

# 6. Calculate loss and update
loss = F.mse_loss(predicted_noise, noise)
loss.backward()
optimizer.step()

By repeating this process millions of times, the U-Net learns to interpret the pose_embedding and use it to transform the appearance information from I_ref into a geometrically correct new view.

Conclusion

Today, we've connected the dots from the theory of camera conditioning to its practical integration in state-of-the-art models. We've seen that generating consistent novel views is a solvable problem, achieved by fine-tuning powerful 2D generative models to be "3D-aware."

Key Takeaways:

• Motivation: Explicit camera conditioning allows diffusion models to overcome the geometric limitations of pure 2D priors, enabling consistent novel view synthesis.
• Architectural Patterns: Camera pose information can be integrated into a U-Net in several ways:
  • Additive Conditioning: Embedding the pose and adding it to the timestep embedding (e.g., SV3D).
  • Cross-Attention: Including the pose vector in the context fed to cross-attention layers (e.g., Zero-1-to-3).
  • Modulation: Using the pose to predict the scale/bias in adaptive normalization layers (e.g., SEVA).
• Leveraging Existing Architectures: We can cleverly repurpose models, like using a video diffusion model's temporal consistency mechanisms for spatial 3D consistency (SV3D).
• The Training Loop: The core process involves training the model on triplets of (reference_image, target_image, relative_pose) rendered from a 3D dataset, using a standard diffusion loss.

Next Lesson Preview:

Now that we can generate consistent views of a single object from any angle, the next logical step is to compose a full scene. In our next lesson, we will apply diffusion models for object placement and spatial arrangement in 3D scenes, exploring how to create complex environments by positioning multiple generated assets.