3D Gaussian Projection for 2D Imaging

Welcome to our next lesson on 3D Gaussian Splatting. In our previous session, we established the fundamental building block of a 3DGS scene: the explicit, anisotropic 3D Gaussian primitive. You learned that each Gaussian is defined by its position, covariance (shape), opacity, and color (represented by Spherical Harmonics). We ended by posing a critical question: how do we transform this 3D collection of ellipsoids into a 2D image?

Today, we will answer that question by diving into the mathematics of the rendering process. Our focus will be on a single primitive. We will derive the complete analytical process for projecting a 3D Gaussian from world space onto a 2D image plane. This procedure is the very origin of the term "splatting" and is central to the efficiency and differentiability of the entire system.

The Core Idea: Analytical Projection

In traditional graphics, we project 3D triangles onto the screen. In 3DGS, we project 3D Gaussians. The key insight that makes this efficient is that the projection of a 3D Gaussian (an ellipsoid) onto a 2D plane is a 2D Gaussian (an ellipse), and this transformation can be calculated analytically. There's a direct mathematical formula to get from the 3D parameters to the 2D parameters.

To build intuition, let's start with a high-level analogy.

This "splatting" process is visualized perfectly in the original paper. The image below shows 3D Gaussians in world space being projected onto an image plane, where they form 2D ellipses. The final image on the right is formed by blending all these ellipses together.

Our goal is to derive the mathematics that govern this projection. We need to find the center, shape, and orientation of the 2D ellipse that results from projecting a given 3D Gaussian.

Step 1: Projecting the Center (Mean)

Let's start with the easy part: the center of the Gaussian. A 3D Gaussian is defined by its mean $\mu$ (a 3D position vector) and its 3x3 covariance matrix $\Sigma$.

Projecting the mean $\mu$ is identical to the standard vertex transformation pipeline you've studied in computer graphics. The 3D world-space coordinate $\mu$ is transformed into a 2D screen-space coordinate using a series of matrix multiplications, typically involving a world-to-camera (view) matrix and a camera-to-screen (projection) matrix.

So, transforming the center is straightforward. The real challenge lies in transforming the shape of the Gaussian, which is defined by the covariance matrix $\Sigma$.

Step 2: Projecting the Covariance

Here is where the process departs from simple vertex transformation. An affine transformation (like rotation or translation) applied to a Gaussian results in another Gaussian whose covariance transforms according to the rule ${\Sigma }^{\prime }=A\Sigma A^T$, where $A$ is the transformation matrix. Our world-to-camera transformation, let's call it $W$, is affine. So, the covariance in camera space is simply ${\Sigma }_{cam}=W\Sigma W^T$.

However, the next step, perspective projection, is not an affine transformation. It involves dividing by the depth coordinate $z$, which makes it non-linear.

Question: Given that perspective projection is non-linear, how can we find the covariance of the transformed Gaussian? We need a way to apply a transformation rule that is only defined for linear (affine) maps. What tool from calculus allows us to find the best local linear approximation of a non-linear function around a specific point?

The answer is the Jacobian matrix. We can approximate the non-linear perspective projection as a linear transformation that is valid in the immediate vicinity of the Gaussian's center. The Jacobian of the projection function, evaluated at the Gaussian's mean in camera space, gives us the matrix for this local linear map.

The following reading provides a great refresher on this concept and sets up the derivation.

Now we have all the pieces. To project the 3D covariance $\Sigma$ into a 2D covariance ${\Sigma }_{2D}$, we follow a two-step affine approximation:

1. World to Camera Space: First, apply the affine world-to-camera view transformation $W$. The covariance in camera space is ${\Sigma }_{cam}=W\Sigma W^T$.
2. Camera to Image Space: Next, apply the local linear approximation of the perspective projection using its Jacobian, $J$.

Combining these, the projected covariance matrix is given by:

The resulting ${\Sigma }^{\prime }$ is a 3x3 matrix. Since we only care about the 2D projection on the image plane (x and y axes), we simply take the upper-left 2x2 submatrix to get our final 2D covariance, ${\Sigma }_{2D}$.

Let's look at a resource that lays out these formulas explicitly.

For someone with your interest in systems, it's also worth noting that this exact equation is what gets implemented in code. The implementation often includes a small fudge factor added to the diagonal of the final 2D covariance matrix to ensure it remains positive semi-definite and invertible, preventing numerical issues during rendering.

Step 3: From 2D Covariance to Ellipse Geometry

We have successfully projected our 3D Gaussian and now have its 2D representation: a projected mean and a 2x2 covariance matrix ${\Sigma }_{2D}$.

What does this ${\Sigma }_{2D}$ geometrically represent? This brings us back to our module on linear algebra. The eigendecomposition of this 2x2 matrix gives us everything we need to draw the ellipse on the screen.

• The two eigenvectors of ${\Sigma }_{2D}$ are orthogonal and point along the major and minor axes of the ellipse, giving us its orientation.
• The corresponding eigenvalues (${\lambda }_1,{\lambda }_2$) are proportional to the squared radii along these axes, giving us the ellipse's size. The radius along an axis is typically taken as $3\sqrt{\lambda }$ to capture a reasonable extent of the Gaussian function.

This final step connects the abstract covariance matrix directly to the visual geometry of the splat.

Conclusion

In this lesson, you have dissected the core mathematical process that underpins 3D Gaussian Splatting. You now understand how a 3D Gaussian primitive is analytically projected onto a 2D image plane to form a "splat."

Key Takeaways:

• Projection Goal: The objective is to find the parameters (center, shape, orientation) of the 2D ellipse that results from projecting a 3D Gaussian.
• Mean Projection: The 3D Gaussian's center (mean) is projected using standard graphics transformations (view and projection matrices).
• Covariance Projection: Because perspective projection is non-linear, we must use its Jacobian matrix $J$ to form a local linear approximation.
• The Full Formula: The 2D covariance ${\Sigma }_{2D}$ is found by computing ${\Sigma }^{\prime }=JW\Sigma W^T{J}^T$ and taking the upper-left 2x2 submatrix, where $W$ is the view matrix.
• Geometric Interpretation: The eigendecomposition of the resulting ${\Sigma }_{2D}$ provides the final geometric properties of the 2D ellipse: the eigenvectors define its orientation, and the eigenvalues define its size.

Preview of the Next Lesson:

We now know how to project a single Gaussian. But a real scene consists of millions of them. How does the renderer handle this massive number of primitives to produce a final image in real time? In our next lesson, we will explore the differentiable rasterization pipeline. We'll cover how these 2D splats are sorted by depth and efficiently blended together on a per-pixel basis to form the final, photorealistic image. This highly optimized pipeline is the final piece of the puzzle to understanding the performance of 3DGS.