Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Large Language Models
·
Module 6
Large-Scale Pre-Training Techniques
Pre-training Data Pipeline: From Collection to Quality Control
Describe the typical pre-training data pipeline, including collection, filtering, deduplication, and quality control.
2
Adam Optimizer for Transformers
Explain the Adam optimizer's use of first and second moment estimates and why it is well-suited for training Transformers.
3
Learning Rate Scheduling for Stable Training
Describe learning rate scheduling strategies, including linear warmup and cosine decay, and their role in stable training.
4
Gradient Accumulation for Larger Batches
Explain gradient accumulation and how it enables training with effectively larger batch sizes on memory-constrained hardware.
5
Mixed-Precision Training: Balancing Speed and Accuracy
Explain mixed-precision training (e.g., bfloat16) and how it reduces memory usage while maintaining numerical stability.
6
Data Parallelism for Large-Scale Model Training
Describe data parallelism and how it enables training on large datasets by replicating the model across multiple devices.
7
Pipeline Parallelism: Scaling Models Across Devices
Describe pipeline parallelism and how it enables training models too large for a single device by splitting layers across devices.
8
Tensor Parallelism: Scaling Large Model Layers
Describe tensor parallelism and how it enables training very large model layers by splitting individual computations across devices.
Previous module
Language Modeling and Training Objectives
Next module
Scaling Laws and Emergent Capabilities