Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Large Language Models
·
Module 5
Language Modeling and Training Objectives
1
Understanding Autoregressive Language Modeling
Define autoregressive (causal) language modeling as sequential next-token prediction, typical for decoder-only models.
2
Masked Language Modeling in BERT
Define masked language modeling as used in BERT and explain how it enables bidirectional context understanding.
3
Softmax & Probability Distributions
Explain how the final linear layer and softmax function produce a probability distribution over the entire vocabulary.
Cross-Entropy Loss from Negative Log-Likelihood
Derive the cross-entropy loss function for next-token prediction from the negative log-likelihood objective.
5
Teacher Forcing: Training vs. Inference
Describe how teacher forcing is used during training and why it differs from autoregressive generation at inference.
Previous module
Foundational Transformer Architectures
Next module
Large-Scale Pre-Training Techniques