MHA, GQA, and MLA: A Comparative Analysis

1. A Tale of Three Architectures

At their core, MHA, GQA, and MLA are all trying to solve the same problem: how to let tokens in a sequence communicate with each other. Their primary difference lies in how they manage the Key (K) and Value (V) vectors, which are stored in the KV cache during inference.

Let's start with a visual comparison of the architectures.

Let's break down what this diagram shows:

• Multi-Head Attention (MHA): This is the original, most expressive architecture. Each of the $H_q$ query heads has its own dedicated pair of key and value heads. This allows for maximum specialization but requires caching $H_q$ separate K/V vectors, leading to a large memory footprint.
• Grouped Query Attention (GQA): This is a practical compromise. The $H_q$ query heads are divided into groups, and all heads within a group share a single pair of K/V heads. This reduces the number of cached K/V vectors from $H_q$ to a smaller number, $H_{kv}$ (the number of groups). Llama 3, for example, uses a group size of 8.
• Multi-head Latent Attention (MLA): This is the most advanced approach, introduced by DeepSeek. Instead of caching full K/V vectors, it caches a single, highly compressed latent vector. During the attention calculation, each query head uses its own learned projection matrix to reconstruct its own specialized K/V vectors "on-the-fly" from this shared latent representation.

To solidify this intuition, the following video provides an excellent animated overview of how MQA (a special case of GQA where there is only one group) and GQA reduce the KV cache, setting the stage for MLA.

2. Trade-off 1: Memory Efficiency

The primary motivation for moving from MHA to GQA or MLA is to reduce the size of the KV cache. The memory cost is directly proportional to the number of key-value heads being stored.

Let's quantify this. The following resource provides a detailed breakdown of the memory costs.

As you saw in the table, the results are striking. For the DeepSeek V3 model architecture:

• An MHA equivalent would require ~30.5 GB of KV cache memory for an 8192-token sequence.
• A GQA equivalent (with a 5-to-1 grouping) cuts this down significantly to ~6.1 GB.
• MLA reduces the memory footprint to a mere ~0.51 GB—a nearly 60x reduction compared to MHA and a 12x reduction compared to GQA.

This dramatic memory saving is the single biggest advantage of MLA, enabling models to handle much longer context windows during inference without running out of GPU memory.

3. Trade-off 2: Compute Cost & Performance

Reducing memory is great, but does it come at the cost of slower computation? This is where the trade-off becomes more nuanced. The analysis involves arithmetic intensity, which is the ratio of computations (FLOPs) to memory accesses (Bytes).

• Memory-Bound Kernels: Have low arithmetic intensity. The GPU spends most of its time waiting for data to be loaded from memory. MHA and GQA attention kernels are typically memory-bound.
• Compute-Bound Kernels: Have high arithmetic intensity. The GPU is saturated with calculations, spending minimal time waiting for data.

Let's dive back into the same article to see how MLA fundamentally changes the nature of the computation.

This analysis reveals a brilliant trade-off made by the DeepSeek team:

• MLA increases the number of raw floating-point operations (FLOPs) in the attention step.
• However, it dramatically decreases the amount of data that needs to be moved from slow HBM memory to the fast on-chip SRAM for those computations.
• The result is a huge increase in arithmetic intensity, which shifts the bottleneck from memory bandwidth to the GPU's computational power. Since modern GPUs are exceptionally good at parallel computation, this makes the overall process significantly faster, as confirmed by the benchmark in Figure 2.

In short, MLA trades a higher volume of cheap, parallelizable compute for a lower volume of expensive memory access, leading to faster inference.

4. Trade-off 3: Model Quality

We've established that MLA is a clear winner in memory efficiency and often in speed. But what about the quality of the model's outputs? Does compressing the K/V information lead to a "dumber" model?

Let's analyze this final trade-off.

The results here are perhaps the most compelling part of the story:

• GQA/MQA: As the table indicates, sharing the exact same K and V vectors across query heads (as GQA/MQA do) leads to a degradation in model quality. The heads lose their ability to specialize.
• MLA: Surprisingly, MLA does not suffer this degradation. In fact, the DeepSeek paper shows that it performs on par with, or even slightly better than, the full MHA baseline.

The intuition is that MLA offers a more flexible form of sharing. Instead of being forced to use identical K/V vectors, each head starts with the same compressed latent information but then uses its own unique up-projection weights ($W_{UK}$, $W_{UV}$) to reconstruct K/V vectors tailored to its specialized function. This retains the expressive power of MHA while gaining the memory efficiency of compression.

Conclusion: Synthesizing the Trade-offs

We have now compared MHA, GQA, and MLA across the three critical axes of memory, compute, and quality.

Key Takeaways:

• MHA is the most expressive but prohibitively expensive in terms of memory for long context lengths.
• GQA is a widely adopted and effective compromise. It significantly reduces memory by sharing K/V heads among groups of query heads, at the cost of a minor drop in model quality.
• MLA represents the state-of-the-art in this domain. It achieves the most significant memory reduction through low-rank compression. By strategically trading increased computation for drastically reduced memory access, it shifts the attention kernel to be compute-bound, resulting in the fastest inference speeds. Crucially, its flexible up-projection mechanism allows it to do all this while maintaining or even improving upon the model quality of full MHA.

Preview of the Next Module:

This concludes our deep dive into efficient attention. We've seen how innovations like MLA have tackled the memory bottleneck in the attention layers. However, the attention layer is only half of a transformer block. The other half is the Feed-Forward Network (FFN), which also presents a scaling challenge. In our next module, we will explore Mixture-of-Experts (MoE), a technique designed to massively scale the number of parameters in the FFN layers without a proportional increase in computational cost.