Pinhole Camera Ray Generation

1. The Geometry of Ray Casting

At its heart, rendering an image of a 3D scene is about figuring out what color each pixel of the image should be. With techniques like NeRF, we do this by casting rays. Imagine your screen is a window looking into a 3D world. For each pixel on that window, we trace a straight line from your eye (the camera), through that pixel, and out into the world. This line is a ray.

Mathematically, a 3D ray is defined by its origin and a direction vector. We can parameterize any point $\mathbf{r}$ along the ray using a distance $t$:

Where:

• $\mathbf{o}$ is the ray origin: the 3D position of the camera in world coordinates.
• $\mathbf{d}$ is the ray direction: a unit vector indicating the direction the ray travels.
• $t$ is the distance from the origin along the ray.

Our goal for this lesson is to compute the specific origin $\mathbf{o}$ and direction $\mathbf{d}$ for the ray corresponding to every single pixel in our desired output image.

To do this, we need two key pieces of information about our virtual camera:

1. Extrinsics: The camera's pose (position and orientation) in the 3D world. This is typically represented by a camera-to-world (c2w) transformation matrix. This matrix directly gives us the ray origin $\mathbf{o}$ and tells us how to orient the ray directions $\mathbf{d}$.
2. Intrinsics: The camera's internal properties, such as focal length and sensor size. These are captured in an intrinsic matrix (K). This matrix allows us to determine the initial direction of the ray in the camera's own local coordinate system before we transform it into the world.

2. The Inverse Transformation: From Pixels to World Space

In a typical graphics pipeline, you start with a 3D point in world space and project it onto a 2D screen. For our task, we need to do the exact opposite: we start with a 2D pixel coordinate and un-project it to find the corresponding 3D ray in world space.

This "un-projection" involves a two-step transformation:

1. Pixel to Camera Space: We use the camera's intrinsic properties to convert the 2D pixel coordinates (i, j) into a 3D direction vector in the camera's local coordinate system.
2. Camera to World Space: We use the camera's extrinsic properties (the c2w matrix) to transform this local direction vector and the camera's origin into the global world coordinate system.

The following video provides an excellent, albeit C++ focused, explanation of this inverse transformation. The underlying mathematics are universal and directly applicable to our goal.

As the video explains, the core idea is to reverse the standard rendering projection. We use the camera's intrinsic parameters to determine where a pixel (i, j) lies on the camera's imaginary image plane, and then use the camera's extrinsic pose to place that ray correctly in the 3D world.

3. A Practical Implementation with NumPy

Now let's translate this theory into a practical Python implementation. The NeRF ecosystem has largely standardized this process, and a clear, concise function is all we need.

The following article provides a well-annotated NumPy implementation named get_rays_np that perfectly captures this logic.

Let's break down the get_rays_np function from the article. It's the embodiment of the two-step process we discussed.

import numpy as np

def get_rays_np(H, W, K, c2w):
    # 1. Create a grid of pixel coordinates
    i, j = np.meshgrid(np.arange(W, dtype=np.float32), 
                       np.arange(H, dtype=np.float32), 
                       indexing='xy')
    
    # 2. Convert pixel coordinates to camera space directions
    #    This uses the intrinsic matrix K = [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
    fx, fy = K[0][0], K[1][1]
    cx, cy = K[0][2], K[1][2]
    dirs = np.stack([(i - cx) / fx, 
                     -(j - cy) / fy, 
                     -np.ones_like(i)], -1)
    
    # 3. Transform camera space directions to world space
    #    This uses the rotation part of the c2w matrix
    #    c2w is a 3x4 matrix: [R | t] where R is 3x3 rotation, t is 3x1 translation
    rotation_matrix = c2w[:3, :3]
    rays_d = np.sum(dirs[..., np.newaxis, :] * rotation_matrix, -1)
    
    # 4. The ray origin is the camera's position in world space
    #    This is the translation part of the c2w matrix
    translation_vector = c2w[:3, -1]
    rays_o = np.broadcast_to(translation_vector, np.shape(rays_d))
    
    return rays_o, rays_d

Dissecting the Code:

• Step 1 (np.meshgrid): This simply creates two 2D arrays, i and j, that hold the x and y coordinates of every pixel in an image of size W x H.
• Step 2 (dirs = ...): This is the Pixel to Camera Space transformation.
  • (i - cx) / fx and (j - cy) / fy convert pixel locations into coordinates on the image plane, which is positioned at a distance z=-1 from the camera center.
  • The -(j - cy) / fy is a common convention because pixel coordinates usually start from the top-left (y increases downwards), whereas in a typical 3D camera coordinate system, the y-axis points upwards.
  • The z-component is set to -1, establishing that we are looking down the negative z-axis in the camera's local frame.
• Step 3 (rays_d = ...): This is the Camera to World Space transformation for the directions. The np.sum(...) line is a vectorized way to multiply every direction vector in dirs by the camera's rotation matrix. This rotates the rays to point in the correct direction in the global 3D world.
• Step 4 (rays_o = ...): This sets the origin for all rays. Since all rays for a single image originate from the same point (the camera's center), we simply take the translation vector from the c2w matrix and broadcast it to match the shape of our rays_d array.

You can test this implementation with a sample set of parameters:

# Example Usage
H, W = 400, 400
focal_length = 555.55 

# Camera Intrinsic Matrix K
K = np.array([
    [focal_length, 0, W/2],
    [0, focal_length, H/2],
    [0, 0, 1]
])

# Camera-to-World Matrix (c2w) - Identity, camera at origin looking down Z
c2w = np.array([
    [1, 0, 0, 0],
    [0, 1, 0, 0],
    [0, 0, 1, 0] 
]) 
# Note: NeRF datasets often use a different convention where Z is up and camera looks forward.
# For example, [0, -1, 0] for Y, [1, 0, 0] for X would make the camera look down +Y.
# We stick to the simpler case here.


rays_o, rays_d = get_rays_np(H, W, K, c2w)

print(f"rays_o shape: {rays_o.shape}")
print(f"rays_d shape: {rays_d.shape}")

# Direction of the ray for the center pixel (200, 200)
# Should be close to [0, 0, -1] since cx,cy = W/2,H/2
center_pixel_direction = rays_d[H//2, W//2]
print(f"Center pixel ray direction: {center_pixel_direction}")

# Direction of the ray for the top-left pixel (0, 0)
top_left_pixel_direction = rays_d[0, 0]
print(f"Top-left pixel ray direction (un-normalized): {top_left_pixel_direction}")

Conclusion

In this lesson, we transitioned from theory to practice, establishing the critical first step of our renderer. We have defined the pinhole camera model, understood the process of un-projecting 2D pixels into 3D space, and analyzed a concrete implementation for generating rays.

Key Takeaways:

• Ray generation is the process of computing a 3D ray, defined by an origin $\mathbf{o}$ and direction $\mathbf{d}$, for each pixel of a target image.
• This process requires two matrices: the intrinsic matrix K (for pixel-to-camera-space mapping) and the extrinsic c2w matrix (for camera-to-world-space transformation).
• The ray origin $\mathbf{o}$ is the camera's position in the world, derived from the translation part of the c2w matrix.
• The ray direction $\mathbf{d}$ is calculated by first finding the direction in the camera's local coordinate system (using K) and then rotating it into the world coordinate system (using the rotation part of c2w).

Next Lesson Preview:

We now know how to generate a grid of rays pointing out into our 3D scene. The next logical step in the volumetric rendering pipeline is to determine the 3D points along these rays that we will evaluate with our neural network. In the next lesson, we will implement stratified sampling of points along a camera ray, taking the rays_o and rays_d we've just created as our starting point.