Scaling Laws and Emergent Capabilities

Explain the Kaplan et al. scaling laws relating model performance (loss) to compute, dataset size, and parameter count.

Explain the Chinchilla scaling laws and how they revised compute-optimal allocation between model size and training data.

Analyze the tradeoffs between model size, dataset size, and compute budget when planning a training run based on scaling laws.

Describe emergent capabilities in large language models and explain how qualitative changes can arise from quantitative increases in scale.

Define in-context learning as an emergent capability and differentiate it from traditional weight-update-based learning (fine-tuning).