Video Diffusion with Temporal Convolutional Networks

From Spatial to Spatiotemporal: The Challenge of Video

A video isn't just a collection of independent images. The primary challenge in video generation is maintaining temporal consistency—ensuring that objects, scenes, and motions evolve coherently from one frame to the next.

A naive approach of generating frames independently would result in a flickering, disjointed mess. The model must understand the relationships between frames. The prevailing strategy to achieve this, and the one we'll focus on, is to adapt powerful, pre-trained 2D image models by inserting new layers designed to operate along the time axis.

Let's begin by reading a brief overview of how 2D U-Nets are commonly extended for video modeling.

As you've just read, the key idea is to factorize operations. Instead of using a full 3D convolution that processes space and time simultaneously, which can be computationally expensive and hard to train, many models separate the operations:

1. Spatial Processing: Apply layers that operate on each frame individually, just like in an image model.
2. Temporal Processing: Insert new layers that operate across frames, sharing information along the time axis.

Let's look at the two primary mechanisms for this temporal processing.

The Building Blocks: Temporal Convolutions and Attention

To give a 2D architecture a sense of time, we need to add layers that explicitly mix information between frames.

1. "Pseudo-3D" Convolutional Layers

This is the core concept for this lesson's learning outcome. Instead of replacing 2D convolutions with full 3D convolutions, we augment them. A "pseudo-3D" convolution block typically consists of:

1. A standard 2D convolution (e.g., Conv2d) that is applied to each frame independently. This captures spatial features within each frame.
2. A 1D convolution (e.g., Conv1d) that is then applied across the temporal dimension. For a given pixel or feature location, this convolution takes inputs from the same location in adjacent frames, effectively mixing information over time.

This factorization is computationally cheaper than a full 3D convolution and allows for the use of powerful pre-trained 2D convolutional layers.

The article from Lilian Weng describes this approach as used in the Make-A-Video model.

2. Temporal Attention Layers

Complementary to convolutions, temporal attention is crucial for capturing long-range dependencies in time. The logic is similar to pseudo-3D convolutions:

1. Spatial Attention: A standard self-attention mechanism is applied to the patches within each frame. During this step, the frames are typically treated as part of the batch dimension.
2. Temporal Attention: A second self-attention mechanism is applied across the frames. Here, for each spatial patch position, the model attends to the corresponding patches at all other timesteps. This allows the model to learn motion and maintain object identity over the duration of the clip.

The right side of the "2D and 1D Temporal Convolutions" image above shows a block diagram of how spatial and temporal attention can be combined.

From Theory to Practice: Implementation Strategies

Now, let's look at how these temporal layers are integrated into a working model. We'll explore two common approaches, both of which you could be called upon to implement as an ML engineer.

Strategy 1: Interleaving Spatial and Temporal Layers in a Transformer

One of the most effective strategies is to take a 2D model architecture and insert temporal blocks after the existing spatial blocks. This is the approach taken by many modern video transformers, which adapt architectures like the Diffusion Transformer (DiT).

The key implementation detail is not a new type of layer, but a clever manipulation of the tensor shapes to reuse standard attention modules. The following video provides an excellent, code-oriented walkthrough.

To summarize the key implementation pattern from the video:

Assume your input tensor x has the shape (batch_size, num_frames, num_patches, hidden_dim).

• For a Spatial Layer (Attention/Conv2D): You want to operate on each frame independently. You can merge the first two dimensions.

  • x_spatial = x.reshape(batch_size * num_frames, num_patches, hidden_dim)
  • output_spatial = spatial_layer(x_spatial)
  • output = output_spatial.reshape(batch_size, num_frames, num_patches, hidden_dim)

• For a Temporal Layer (Attention/Conv1D): You want to operate across frames for each patch. You can permute the dimensions to make the num_frames dimension the one the layer operates on.

  • x_temporal = x.permute(0, 2, 1, 3).reshape(batch_size * num_patches, num_frames, hidden_dim)
  • output_temporal = temporal_layer(x_temporal)
  • output = output_temporal.reshape(batch_size, num_patches, num_frames, hidden_dim).permute(0, 2, 1, 3)

This pattern of reshape -> apply layer -> reshape is a highly efficient way to extend 2D models for spatiotemporal tasks.

Strategy 2: Building a Native 3D U-Net

An alternative is to build the U-Net with 3D operations from the ground up, using PyTorch's nn.Conv3d layers. This approach directly builds a network that expects 5D tensors (B, C, F, H, W).

The following code repository implements a video generation model using a full 3D U-Net.

This 3D U-Net approach is more direct, but it doesn't allow for easily leveraging pre-trained weights from a 2D image model, which is a major advantage of the "interleaving" strategy. In practice, many state-of-the-art models use a hybrid, often inserting 1D temporal convolutions and attention layers into a pre-trained 2D U-Net, effectively combining the best of both strategies.

class ResnetBlock2D(nn.Module):
    def forward(self, x, time_emb):
        h = self.norm1(x)
        h = self.act1(h)
        h = self.conv1(h)
        
        # ... more layers ...
        
        h = self.conv2(h)
        return h + x

Conclusion

In this lesson, we took our first and most critical step into video generation. We established that extending 2D models is a powerful and common paradigm, and we investigated the primary architectural tools for doing so.

Key Takeaways:

• Video generation's main challenge is maintaining temporal consistency.
• We can "inflate" a 2D architecture by inserting layers that operate across the time dimension.
• Pseudo-3D convolutions factorize a 3D operation into a spatial Conv2d followed by a temporal Conv1d.
• Temporal attention allows for capturing long-range relationships between frames.
• Implementation often relies on clever tensor reshaping and permuting to apply 2D or 1D layers along the correct dimensions of a 5D spatiotemporal tensor (B, C, F, H, W).

Next Lesson Preview:

We have now added the basic anatomical parts for processing time to our model. However, the temporal consistency we get from these layers is implicit—learned by the model from data. In the next lesson, we will explore a more explicit method for enforcing coherence by implementing a frame-to-frame consistency technique that conditions the generation of each new frame on the frame that came immediately before it.