Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Elite ML Engineer from Day 1
·
Module 5
Training and Scaling Large Language Models
1
Tokenization Methods: A Comparative Analysis
Describe and compare tokenization methods (BPE, WordPiece, SentencePiece) and their trade-offs.
2
Building a BPE Tokenizer
Implement a basic Byte-Pair Encoding (BPE) tokenizer to understand vocabulary construction and subword segmentation.
3
Scaling Laws in Machine Learning
Explain the relationship between model size, dataset size, and performance using scaling laws.
4
Chinchilla Scaling Laws: Estimating Training Compute (FLOPs)
Estimate training compute requirements (FLOPs) for a given model and dataset size using Chinchilla scaling laws.
5
Mixed-Precision Training: Accelerating Training and Reducing Memory
Explain the role of mixed-precision training (FP16, BF16) in accelerating training and reducing memory usage.
6
Gradient Accumulation for Larger Batches
Explain gradient accumulation as a technique to simulate larger batch sizes with limited hardware.
7
LLM Distributed Training Paradigms
Compare distributed training paradigms for LLMs: Data Parallelism, Tensor Parallelism, and Pipeline Parallelism.
Previous module
Modern LLM Architectures and Optimizations
Next module
Prompt Engineering Fundamentals