Skip to main content
grasp.study
About
Explore
Pricing
Log in
Try it for free
Back to course
👋
Welcome to a Grasp lesson
Create a free account
to keep studying and create your own study path.
Elite ML Engineer from Day 1
·
Module 4
Modern LLM Architectures and Optimizations
1
Mixture-of-Experts: Scaling Models Efficiently
Describe the Mixture-of-Experts (MoE) architecture and its benefits for scaling model size while maintaining computational efficiency.
2
Understanding Rotary Positional Embeddings (RoPE)
Explain Rotary Positional Embeddings (RoPE) and their advantages over absolute or learned positional encodings.
3
KV-Cache: Accelerating Autoregressive Inference
Explain the function of the KV-cache in accelerating autoregressive inference.
4
Optimizing Attention: GQA & MQA Explained
Describe Grouped-Query Attention (GQA) and Multi-Query Attention (MQA) as optimizations for the standard multi-head attention.
5
Comparing LLM Activation Functions: SwiGLU vs. GeLU
Compare different activation functions used in modern LLMs, such as SwiGLU and GeLU.
6
LLM Architecture: Open-weight vs. Proprietary Models
Analyze and compare the architectural choices (e.g., MoE, GQA, attention window) of leading open-weight (LLaMA 3, Mistral) and proprietary (GPT-4, Claude 3) models.
7
From Transformers to LLMs: An Architectural Journey
Trace the architectural evolution from the original Transformer to modern LLMs, identifying key trends and innovations.
Previous module
Attention Mechanisms and Architectural Variants
Next module
Training and Scaling Large Language Models