Deep Learning for Transformers

Create PyTorch tensors and interpret their batch, sequence, and feature dimensions.

Reshape and transpose tensors while tracking how their dimensions change.

Predict the result shape of a broadcasted PyTorch operation.

Express a fully connected layer as an affine transformation.

Trace data through a two-layer feedforward neural network.

Use PyTorch autograd to obtain gradients of a scalar loss.

Implement one PyTorch training step with loss calculation, backpropagation, and a parameter update.

Map the components of a small PyTorch neural network to their corresponding mathematical operations.

Convert a sentence into token IDs using a subword tokenizer.

Retrieve token embeddings from an embedding matrix and determine the resulting tensor shape.

Formulate language modeling as next-token prediction.

Construct shifted decoder inputs and targets for teacher-forced training.

Convert a decoder hidden state into vocabulary probabilities with a linear projection and softmax.

Trace information through a conventional encoder-decoder sequence model.

Trace the hidden-state recurrence of a simple recurrent neural network.

Explain how recurrence creates sequential computation and long information paths.

Distinguish the roles of queries, keys, and values in an attention operation.

Compute query-key similarity scores for a small attention example.

Apply the square-root key-dimension scaling factor to attention scores.

Convert attention scores into attention weights with softmax.

Compute an attention output as a weighted sum of value vectors.

Apply a padding mask to prevent attention to padded tokens.

Apply a causal mask to prevent attention to future tokens.

Implement single-head scaled dot-product attention using PyTorch tensor operations.

Determine the tensor shapes created by the query, key, and value projections in multi-head attention.

Combine multiple attention heads by concatenation and an output projection.

Explain how independent attention heads can represent different token relationships.

Trace information through a residual connection around a sublayer.

Apply layer normalization to a token representation.

Compute the output of a position-wise feedforward network for one token.

Calculate sinusoidal positional-encoding components for a specified token position.

Trace the sublayers and data flow of the original Transformer encoder and decoder.

Use the paper’s complexity table to compare the per-layer cost of self-attention, recurrence, and convolution.

Compare the amount of sequential computation required by self-attention and recurrent layers.

Compare the maximum information-path lengths of self-attention, recurrent, and convolutional layers.

Explain the purpose of the Transformer’s learning-rate warmup and inverse-square-root decay.

Identify where dropout is applied in the original Transformer.

Explain how label smoothing changes the model’s training targets.

Interpret the paper’s BLEU results without treating BLEU as a direct measure of human understanding.

Draw a supported conclusion from one of the paper’s ablation studies.

Reconstruct the major components and connections in the paper’s Transformer architecture diagram from memory.

Trace a source sentence and shifted target sentence through the complete Transformer data flow.

Calculate the principal tensor shapes in an end-to-end Transformer example.

Implement the forward pass of a compact Transformer encoder layer in PyTorch.

Distinguish the paper’s base and big model configurations at a conceptual level.

State which experimental findings support the paper’s central claims and which claims remain assumptions.

Diagnose common misconceptions about attention weights, parallelism, positional encoding, and autoregressive decoding.

Deliver a structured five-minute viva explanation of why the Transformer can replace recurrence for sequence transduction.