Recommended resources
The freely available resources this lesson will direct you to.
- Stanford Seminar - Nvidia’s H100 GPUwww.youtube.com
- NVIDIA GTC May 2020 Keynote Pt6: NVIDIA A100 Data Center GPU Based on NVIDIA Ampere Architecturewww.youtube.com
- 1. NVIDIA Blackwell Tuning Guide — Blackwell Tuning Guide 13.4 documentationdocs.nvidia.com
- Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era | NVIDIA Technical Blogdeveloper.nvidia.com
Hello! Last lesson estimated the sectors a kernel requests and kept that estimate separate from bytes actually transferred to device memory. That distinction becomes useful when reading a new GPU specification: a larger cache, faster HBM, or new execution unit matters only if it changes a resource your workload uses.
This lesson develops a way to extract performance hypotheses, rather than speedup promises, from public architecture documentation. We will use Hopper as a baseline and Blackwell-family documents as a case study. The aim is a comparison you could explain and defend without relying on nonpublic implementation details.
Establish the baseline before calling something new
Architecture announcements often group several improvements together. The first question is not “What does the new chip contain?” but “What changed relative to which predecessor?” A capability inherited from an earlier generation can still be important, but it is not evidence of a new-generation speedup.
For a concise Hopper baseline, watch these excerpts from Stanford Seminar: H100 GPU, published by Stanford Online.
Stanford Seminar - Nvidia’s H100 GPU
Watch this seminar to identify capabilities that Blackwell inherits, rather than introduces.
Watch the SM overview for Hopper's unified L1/shared-memory capacity and Tensor Memory Accelerator (TMA). Then watch block clusters for the purpose of clusters and distributed shared memory. Focus on which features already existed before Blackwell; you do not need to retain every implementation detail.
Two immediate corrections to a hurried “what’s new” list follow. TMA and thread-block clusters are not Blackwell inventions. Nor is a 256 KB combined L1/texture/shared-memory pool, by itself, evidence that this capacity increased from Hopper. To find a genuine change, compare the same feature at the same level: per SM with per SM, chip capacity with chip capacity, and a particular product configuration with another particular product configuration.
A useful comparison record has five fields: product and compute capability; baseline; documented change; affected workload; measurement that could test the effect. Leave a field marked unknown when the document does not establish it. That is more informative than quietly substituting a specification from a different product.
Read limits as constraints, not throughput claims
The Blackwell Tuning Guide in the official CUDA documentation is a good source for programmable limits. Read the specified subsections, paying particular attention to their product and compute-capability qualifiers.
1. NVIDIA Blackwell Tuning Guide — Blackwell Tuning Guide 13.4 documentation
Read the official CUDA tuning guide for limits that can change a kernel's feasible launch configuration, and for memory-system details that require careful product attribution.
In Section 1.4.1.1, “Occupancy,” read the resource limits, beginning with concurrent warps and ending with per-block shared memory. In Section 1.4.1.2, “Thread Block Clusters,” go to the paragraph beginning “The maximum portable cluster size supported is 8” and read the cluster limits through its warning about active blocks. In Sections 1.4.2.2, “Increased L2 Capacity,” and 1.4.2.3, “Unified Shared Memory/L1/Texture Cache,” read the capacity details, from the GB200 L2 statement through the B200 per-block allocation explanation. Continue in Section 1.4.2.3 with the cache description. Note each time the text says B200, GB200, compute capability 10.0, or compute capability 12.0.
The guide specifies 64 resident warps per SM for compute capability 10.0 and 48 for 12.0, while giving both a 64K-register file of 32-bit registers per SM. Those are limits on potential residency, not promises that either device will execute a given kernel faster.
For example, imagine 256-thread blocks using 32 registers per thread and negligible shared memory. Each block has eight warps and requests 8,192 registers. Ignoring allocation granularity and other constraints, the register file could accommodate eight such blocks. The warp limit permits at most eight on a 10.0 SM but six on a 12.0 SM. That calculation identifies a possible residency difference; it does not establish their relative instruction throughput, latency, or runtime. A kernel with different register or shared-memory requirements could encounter another limit first.
Other statements need equally narrow labels. The guide assigns 126 MB of L2 to GB200; it does not establish that every Blackwell-family product has 126 MB. It describes the B200 combined L1/texture/shared-memory capacity as the same 256 KB maximum as Hopper, not an increase over Hopper. And although a B200 block can address up to 227 KB of shared memory with the required opt-in, a larger tile might reduce the number of blocks that fit concurrently.
Cluster size illustrates why supported and advantageous are different claims. Eight is the maximum portable cluster size stated in the guide; B200 can opt in to a nonportable size of 16. More cooperating blocks may enable useful data sharing, but larger clusters can also constrain active blocks across the GPU. The specification identifies an option and a trade-off, not a universal tuning recommendation.
Translate each feature into a workload condition
A chip overview helps locate changes the tuning guide does not describe in depth. Read the selected portions of Inside Blackwell Ultra: The Chip Powering the AI Factory Era, from a vendor technical blog. Treat its peak figures as product-scoped specifications, not measured results for your kernel.
Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era | NVIDIA Technical Blog
Read the technical blog for Blackwell Ultra's SM, attention, memory, and interconnect claims. Look for the workload condition attached to each claimed benefit.
In “Streaming multiprocessors: compute engines for the AI Factory,” begin where an SM “is a self-contained compute engine” and read the SM components through the SFU bullet. In “Accelerated softmax in the attention layer,” read the SFU claim; identify which instructions, rather than which whole workload, gained throughput. In “Memory: high capacity and bandwidth for multi-trillion-parameter models,” begin at “With 288 GB of HBM3e per GPU” and read the HBM figures through the bandwidth bullet. In “Interconnect: built for scale,” read the link context, then inspect Table 1 immediately below it. Keep the table's bidirectional bandwidth convention in mind.
The SM diagram makes one distinction especially easy to miss:
The blog describes 256 KB of TMEM per SM, illustrated as four 64 KB portions. It also depicts a 256 KB L1-data-cache/shared-memory pool. Seeing “256 KB” twice does not mean a CUDA kernel receives 512 KB of ordinary shared memory. TMEM is associated with particular Tensor Core operations; whether it reduces traffic for a workload depends on the operations and implementation that use it.
A compact way to turn the reading into hypotheses is to ask what must be true before a feature can help:
| Documented feature | Plausible beneficiary | Conclusion the specification alone cannot support |
|---|---|---|
| Fifth-generation Tensor Cores and newer low-precision formats | Matrix operations that actually use supported formats and instructions | Ordinary FP32 code speeds up by the quoted low-precision factor |
| Higher throughput for selected Ultra SFU instructions | A stage substantially limited by those instructions | An entire attention kernel becomes twice as fast |
| Larger Ultra HBM capacity | A working set that otherwise needs offloading or a different layout | A resident streaming kernel gains bandwidth from capacity alone |
| Faster GPU-to-GPU interconnect | Communication using the relevant links and topology | Every multi-GPU application sees the link’s peak ratio |
Notice the baseline changes across those rows. The blog compares Ultra’s SFU improvement with earlier Blackwell, while it compares Ultra’s 8 TB/s HBM figure with an H100 figure of 3.35 TB/s. It also gives 8 TB/s for the reference Blackwell configuration. Thus “Ultra has more HBM capacity” and “Ultra has more HBM bandwidth than that Blackwell configuration” are not equivalent statements. Product, comparison point, and metric must travel together.
Turn a specification into a falsifiable prediction
Suppose you are assessing a streaming kernel on H100 and Blackwell Ultra. From the previous lesson, you may know its logical bytes and estimated sector demand. Neither is automatically its DRAM traffic. To use an HBM bandwidth figure, first stipulate or measure bytes crossing the device-memory boundary.
As a deliberately idealized example, assume the kernel transfers 8 GB at that boundary on each device, with no change in the amount of work. Dividing by the cited peak bandwidths gives lower bounds on time spent transferring those bytes:
The bandwidth ratio is about 2.4. It is not a predicted kernel speedup. Achieved bandwidth can fall below peak; the kernel may also spend time on instructions, dependencies, or other memory levels. A change in cache behavior might alter the assumed 8 GB itself. The defensible prediction is narrower: if this workload is limited by sustained HBM bandwidth and transfers comparable DRAM bytes, the higher peak raises its performance ceiling. A controlled benchmark and boundary-appropriate traffic measurement could test that claim.
Now change the workload while keeping the product comparison:
- For a working set that fits in one product’s usable cache but not another’s, cache capacity may matter more than peak HBM bandwidth. A stated cache size alone cannot establish residency or hit rate.
- For matrix math, check the actual operand format and instruction path before applying a Tensor Core throughput figure. Comparing FP8 throughput on one device with NVFP4 throughput on another would also change the numerical workload.
- For communication, retain whether the quoted figure is bidirectional aggregate link bandwidth, and check which links the transfer actually traverses. It is not interchangeable with one-direction application payload throughput.
These are different predictions because they identify different limiting resources. Resist turning a product comparison table into one undifferentiated “generation speedup.”
Keep an architecture-change ledger
When reading documentation for any future generation, make a short ledger rather than a list of impressive numbers. One entry might read:
Claim: Blackwell Ultra has higher throughput for selected SFU instructions than the reference Blackwell GPU. Scope: those instructions on those product configurations. Potential effect: less time in a stage limited by their issue throughput. Unknowns: generated instruction mix, dependencies, and the stage’s share of total runtime. Test: compare matched implementations and inspect instruction and timing evidence.
For each entry, check four things:
- Provenance and scope. Is this a programming limit, peak chip specification, or application result? Which SKU, compute capability, precision, and direction of transfer does it describe?
- A real baseline. Did the predecessor already have the feature? Are the two numbers defined in the same way?
- A causal workload condition. What resource must the kernel stress for this change to matter, and what competing limit might hide the benefit?
- Observable evidence. Which timing or memory-boundary measurement would distinguish the hypothesis from an appealing but unsupported story?
This discipline is particularly useful in profiling work: a new hardware block or changed capacity may call for new observability, but a public diagram does not establish counter definitions or comparability across generations. Keep those questions marked as open until supported by documentation and measurement.
Takeaways
Extracting an architectural change means doing more than copying a specification. Establish the predecessor, preserve the product and metric scope, identify the workload condition under which the change could matter, and state what would test it. Hopper’s existing clusters and TMA, Blackwell’s compute-capability-specific residency limits, GB200’s product-specific L2 figure, and Blackwell Ultra’s distinct TMEM and shared-memory resources show why those qualifiers matter.
This closes the architecture-as-performance-model module. Next, you will build a benchmark harness that separates kernel execution from allocation and transfers, giving these documentation-based hypotheses a trustworthy first measurement.