GPU Memory Management & Data Transfer

1. The Host-Device Memory Distinction

As we discussed, the CPU and GPU are separate processors, and in the most common hardware configuration, they have their own physically distinct memory spaces. The host uses system RAM, while the device uses its own onboard video RAM (VRAM or Global Memory).

Because of this separation, a pointer created on the host (e.g., via malloc or new) is not valid on the device, and vice-versa. Your C++ experience with pointers and memory allocation is directly relevant here, but now we must manage two separate memory pools and orchestrate the data movement between them.

2. Allocating and Freeing Device Memory

To allocate memory on the device, we use CUDA's equivalent of the standard C malloc function: cudaMalloc.

cudaError_t cudaMalloc(void** devPtr, size_t size);

Let's break down its parameters:

• devPtr: This is a pointer to a pointer. cudaMalloc will allocate size bytes of memory on the device and then write the address of that allocated memory into the pointer variable that devPtr points to. Because you're comfortable with C++, you can recognize this pattern: the function needs to modify the value of a pointer in the calling scope, so we pass its address.
• size: The number of bytes to allocate.

To free the memory, we use the corresponding cudaFree function, which is analogous to free:

cudaError_t cudaFree(void* devPtr);

Here, devPtr is the pointer to the device memory you wish to deallocate.

For a formal summary of these functions, please review the following resource.

3. Transferring Data Between Host and Device

Once you have allocated memory on both the host and the device, you need a way to copy data between them. The primary function for this is cudaMemcpy.

cudaError_t cudaMemcpy(void* dst, const void* src, size_t count, cudaMemcpyKind kind);

The parameters are:

• dst: A pointer to the destination memory location.
• src: A pointer to the source memory location.
• count: The number of bytes to copy.
• kind: A flag that specifies the direction of the copy. This is crucial.

The kind parameter can be one of the following:

• cudaMemcpyHostToDevice: Copies from host memory to device memory.
• cudaMemcpyDeviceToHost: Copies from device memory back to host memory.
• cudaMemcpyDeviceToDevice: Copies from one location in device memory to another.

It's important to know that cudaMemcpy is a synchronous (or blocking) function. When your host code calls cudaMemcpy, the host thread will pause and wait until the entire memory transfer is complete before executing the next line of code.

Let's refer back to the presentation slides for a clear definition of this function.

4. Code Walkthrough: The Full Memory Management Flow

Now, let's see these functions in the context of a complete program. We will walk through a vector addition example, focusing only on the host-side code that manages memory. The following video demonstrates this entire process.

To reinforce this, the video below offers a second walkthrough. The style is slightly different, but it covers the exact same fundamental steps of allocating GPU memory and copying data to and from it.

5. Quick Practice

Based on what you've just learned, consider how you would set up the memory for a program that adds two vectors, A and B (each of size N), and stores the result in a vector C.

On a piece of paper or in a text editor, write out the C++/CUDA code snippet to perform the following steps:

1. Assume host pointers float *h_a, *h_b, *h_c; and device pointers float *d_a, *d_b, *d_c; have been declared.
2. Assume h_a and h_b have been allocated and initialized on the host. Allocate memory for h_c.
3. Allocate the corresponding memory on the device for d_a, d_b, and d_c.
4. Copy the contents of h_a and h_b to d_a and d_b, respectively.
5. (Imagine the kernel runs here)
6. Copy the resulting vector from d_c back to h_c.
7. Free all the allocated device memory.

This exercise will help solidify the sequence of operations.

// 1. Pointers are declared
float *h_a, *h_b, *h_c;
float *d_a, *d_b, *d_c;
int N = 1024;
size_t bytes = N * sizeof(float);

// 2. Allocate and initialize host memory
h_a = (float*)malloc(bytes);
h_b = (float*)malloc(bytes);
h_c = (float*)malloc(bytes);
// ... code to fill h_a and h_b with data ...

// 3. Allocate device memory
cudaMalloc((void**)&d_a, bytes);
cudaMalloc((void**)&d_b, bytes);
cudaMalloc((void**)&d_c, bytes);

// 4. Copy input data from host to device
cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice);
cudaMemcpy(d_b, h_b, bytes, cudaMemcpyHostToDevice);

// 5. (Kernel launch would go here)
// vectorAdd_kernel<<<...>>>(d_a, d_b, d_c, N);

// 6. Copy result from device to host
cudaMemcpy(h_c, d_c, bytes, cudaMemcpyDeviceToHost);

// 7. Free device memory
cudaFree(d_a);
cudaFree(d_b);
cudaFree(d_c);

// Finally, free host memory
free(h_a);
free(h_b);
free(h_c);

Conclusion

In this lesson, we moved from the conceptual model of host-device interaction to the concrete functions that make it happen. You now have the tools to manage the lifecycle of data in a CUDA application.

Key Takeaways:

• Separate Memory Spaces: The host (CPU) and device (GPU) have their own memory, requiring explicit data management.
• Memory Allocation: Use cudaMalloc() to allocate memory on the device and cudaFree() to release it. Remember the void** parameter in cudaMalloc is used to modify the host's pointer variable.
• Data Transfer: Use cudaMemcpy() to move data between the host and device. The cudaMemcpyKind flag is essential for specifying the direction of the transfer.
• Standard Workflow: The sequence of allocating on the device, copying inputs, launching a kernel, copying results, and finally freeing device memory is the backbone of most CUDA programs.

In our next lesson, we will finally dive into step 3: writing and launching a basic CUDA kernel. We'll see how to write the __global__ function that runs on the GPU and learn the <<<...>>> syntax to launch it from the host, bringing all the pieces together to perform a complete parallel computation.