Transformer Architecture Fundamentals

Describe the high-level encoder-decoder structure of the original Transformer and its core components.

Implement the scaled dot-product attention mechanism in PyTorch.

Implement multi-head attention and explain how it enables learning different representation subspaces.

Describe the role and implementation of absolute positional encodings.

Explain the purpose of the position-wise feed-forward network in each transformer block.

Implement layer normalization and explain its importance for stabilizing training in deep transformers.

Trace the data flow through a full encoder and decoder block, including all sub-layers and residual connections.

Calculate the number of parameters in a transformer model given its configuration (layers, heads, dimensions).