The Attention Mechanism

Compute scaled dot-product attention given query, key, and value matrices

Explain the role of the scaling factor in preventing vanishing gradients through softmax

Describe self-attention and how it allows each token to attend to all other tokens in a sequence

Explain causal (masked) self-attention and why it is required for autoregressive models

Describe multi-head attention and how it enables attending to different representation subspaces simultaneously

Explain the role of learned projection matrices W_Q, W_K, W_V, and W_O in multi-head attention

Describe sinusoidal positional encodings and why position information must be explicitly injected