Sequence Modeling Foundations

Describe how tokenization converts text into sequences of integer token IDs

Explain how embedding layers map token IDs to dense vector representations

Define autoregressive language modeling as the task of predicting the next token given all previous tokens

Compute the softmax function to convert raw logits into a probability distribution over the vocabulary

Compute the cross-entropy loss for a language model's next-token predictions