The Core Attention Mechanism

Define the roles of query, key, and value vectors in the attention mechanism.

Explain how dot products between queries and keys measure token relevance.

Compute scaled dot-product attention given Q, K, and V matrices, including the application of softmax to convert scores into a probability distribution.

Explain why the dot products are scaled by the square root of the key dimension to prevent softmax saturation and vanishing gradients.

Explain how causal masking is implemented to prevent attention to future tokens in autoregressive models.