Recommended resources
The freely available resources this lesson will direct you to.
Introduction
In our last lesson, we learned how to interpret profiler traces to diagnose whether a workload is compute-bound, memory-bound, or communication-bound by looking for patterns like GPU idle time or dominant communication kernels. This gave us a methodology for identifying the type of bottleneck we're facing.
This lesson builds directly on that by examining the tools themselves. We will address the learning outcome: Describe the capabilities of system-level profilers (e.g., Nsight Systems) vs. framework-level profilers (e.g., PyTorch Profiler).
The key insight is that profiling is often a multi-stage process, moving from a high-level, application-centric view to a low-level, hardware-centric one. Different tools are optimized for different stages of this process. By the end of this lesson, you will understand the distinct roles of framework-level and system-level profilers and how to use them in a complementary workflow to pinpoint performance issues, answering not just "what is slow?" but also "where is it in my code?" and "why is it slow?".
1. The Framework-Level View: Answering "What" and "Where" in Your Code
The first step in profiling is usually to understand performance in the context of your application's logic. This is the domain of framework-level profilers like the PyTorch Profiler.
These tools are integrated directly into the ML framework (e.g., PyTorch, TensorFlow). Their primary strength is their awareness of the framework's abstractions. They understand concepts like model layers, autograd, and optimizer steps.
The key capability of the PyTorch Profiler is its ability to correlate low-level GPU activity (CUDA kernels) back to the high-level PyTorch operations that generated them.
Lecture 16: On Hands Profiling
To see this distinction in action, let's watch a segment from the 'On Hands Profiling' lecture by Taylor, an engineer formerly on the PyTorch team. He will first show a trace from a system-level profiler and then contrast it with one from the PyTorch Profiler.
Please watch from 04:38 to 09:57. As you watch, focus on: At the beginning (04:38-06:27), notice how the Nsight Systems trace shows GPU kernels like cutlass and flash_backward, but the presenter notes it's 'very difficult to go from this kind of view to something prescriptive'. When he switches to the PyTorch Profiler (06:27-09:08), observe the key difference: the ability to select a kernel and walk up the call stack to the exact line of Python source code that triggered it. The final segment (09:08-09:57) provides a direct comparison, highlighting how the PyTorch Profiler helps find 'gaps' in execution and relate them to specific model layers like RMSNorm.
As the video demonstrates, the PyTorch Profiler's value is in bridging the gap between hardware execution and your source code. It answers questions like:
- Which
nn.Moduleis consuming the most GPU time? - How much time is spent in the forward pass vs. the backward pass vs. the optimizer step?
- Are there significant delays caused by CPU-side Python logic between GPU operations?
- How much memory is allocated by each operation?
It provides this information through visual timeline traces (in Chrome/Perfetto format) and summary tables printed to the console. However, while it tells you what PyTorch operation is slow, it provides limited information on why the underlying CUDA kernels are performing poorly at the hardware level.
2. The System-Level View: Answering "How" the System Behaves
When you need to understand the broader interactions between your application and the system, you turn to a system-level profiler like NVIDIA Nsight Systems.
Unlike the PyTorch Profiler, Nsight Systems is framework-agnostic. It captures a holistic timeline of events across the entire system, including:
- CPU core activity (including context switches and scheduling).
- GPU activity (kernel execution, memory copies).
- CUDA API calls made by the application.
- Data transfers across interconnects like PCIe and NVLink.
- OS runtime library events.
This system-wide view is crucial for diagnosing issues that are not visible from within the Python framework alone.
Intro to NVIDIA Nsight Systems | CUDA Developer Tools
This brief introduction from NVIDIA Developer provides a concise overview of what Nsight Systems is designed for.
Please watch the following two clips: 00:11 - 00:58: This introduces Nsight Systems' goal: to create a 'holistic, easy-to-understand timeline view of the whole system' to pinpoint issues like slow kernels, CPU bottlenecks, or PCIe transfers. 06:31 - 08:28: This section details the timeline view itself. Pay attention to how it separates CPU thread activity (like cudaMemcpy calls) from the actual hardware events on the GPU (kernel execution and data transfers). This separation is key to understanding CPU-GPU interaction.
The primary strength of Nsight Systems is in diagnosing systemic issues. It helps answer questions like:
- Is my GPU idle because the CPU is not submitting kernels fast enough (a "CPU-bound" scenario)?
- Are data transfers between the CPU and GPU happening efficiently? Are they serialized or overlapping with computation?
- Are my CUDA streams executing concurrently to hide latency?
- In a multi-GPU setting, how are communication and computation overlapping?
However, as we saw in the first video, a raw Nsight Systems trace can be difficult to interpret because the kernels are anonymous from the perspective of your high-level code. To bridge this gap, we use the NVIDIA Tools Extension (NVTX). You can insert NVTX "ranges" into your Python code to label specific sections (e.g., forward_pass, optimizer_step). These labels then appear directly in the Nsight Systems timeline, making it far more readable.
3. A Multi-Layered Profiling Workflow
Framework-level and system-level profilers are not competitors; they are complementary tools used in a "macro-to-micro" workflow. A third category, kernel-level profilers like NVIDIA Nsight Compute, completes this picture by providing deep microarchitectural analysis of a single kernel.
The following table and workflow summarize how these tools fit together.
Fix GPU Bottlenecks: PyTorch Profiler + Nsight
The article 'Fix GPU Bottlenecks' from AceCloud provides an excellent comparison table and a practical workflow that aligns with this multi-layered approach. The research paper 'Profiling & Monitoring Deep Learning Training Tasks' also formalizes this idea.
Please read the section 'When Should You Use Nsight Compute vs PyTorch Profiler?'. Focus on the comparison table and the 'Key Takeaways', which crisply define the roles of each tool. Note that the article uses 'Nsight Compute' as the low-level example, which we'll place as the final step in our workflow.
Let's synthesize this into a clear comparison.
| Feature | PyTorch Profiler (Framework-Level) | NVIDIA Nsight Systems (System-Level) | NVIDIA Nsight Compute (Kernel-Level) |
|---|---|---|---|
| Primary Focus | PyTorch operations, Python code, autograd | System-wide events, CPU-GPU interaction, API calls | Individual CUDA kernel microarchitecture |
| Key Question | What part of my model is slow? | How is my application interacting with the system? | Why is this specific kernel slow? |
| Level of Detail | High-level (e.g., nn.Linear, source code lines) |
Mid-level (e.g., CUDA API calls, kernel streams, PCIe transfers) | Low-level (e.g., warp stall reasons, cache hit rates, memory throughput, occupancy) |
| Use Case | First step; finding bottlenecks in the application logic. | Diagnosing CPU waits, I/O bottlenecks, inefficient data movement, or stream concurrency issues. | Final step; deep-diving into a "hot" kernel to optimize its performance. |
| Overhead | Moderate; integrated into Python. | Low; runs as a separate process collecting system traces. | Very high; often replays the kernel multiple times to collect hardware counter data. |
This leads to a standard, highly effective profiling workflow:
- Start with the PyTorch Profiler: Use it to get a macro view of your training step. Identify the most time-consuming operations in the context of your PyTorch code. Is a specific layer slow? Is the data loader the bottleneck? This answers the "what" and "where".
- Analyze with Nsight Systems: If you see large GPU idle gaps or suspect inefficient data movement, use Nsight Systems to get a system-wide view. By adding NVTX ranges to your code, you can correlate system events with your application phases. This helps diagnose the "how".
- Optimize with Nsight Compute: If the PyTorch Profiler points to a specific, long-running kernel as the bottleneck (and Nsight Systems shows it's not simply waiting on data), use Nsight Compute to perform a deep microarchitectural analysis. This will tell you why that kernel is slow (e.g., it's memory-bound due to high-latency memory fetches), guiding targeted optimizations.
As the paper "Profiling & Monitoring Deep Learning Training Tasks" states, "...a framework-based or system-wide GPU profiler can first detect the frequent kernels or bottlenecks, and then, a lower-level GPU profiler can focus on particular kernels at the micro-architectural-level."
Finally, it's worth noting that system-level profilers like Nsight Systems are indispensable for diagnosing performance in distributed environments, a key area of your interest. They can profile multiple nodes simultaneously to visualize communication patterns and identify stragglers.
Conclusion
In this lesson, we differentiated between framework-level and system-level profilers, understanding them as complementary tools in a unified workflow.
Key Takeaways:
- PyTorch Profiler (Framework-Level): Is application-aware. Its strength is mapping GPU work back to your PyTorch source code. It tells you what part of your model is slow.
- NVIDIA Nsight Systems (System-Level): Is system-aware. Its strength is providing a holistic timeline of CPU-GPU interactions, API calls, and data movement. It explains how your application interacts with the hardware.
- NVIDIA Nsight Compute (Kernel-Level): Is hardware-aware. Its strength is deep microarchitectural analysis of a single kernel. It explains why a specific kernel is slow.
- A standard workflow is to start high with the PyTorch Profiler to identify hot spots in your code, then use Nsight Systems to check for system-level issues, and finally use Nsight Compute to optimize critical kernels.
Preview of the Next Lesson:
Now that we understand the different types of profilers and their roles, we are ready to apply this knowledge. In the next lesson, Analyze a profiler trace to identify performance limiters, we will work through a practical example, interpreting a real profiler trace to diagnose a performance issue and suggest optimizations.