Optimization for Deep Networks

Define convex sets and convex functions, and use the Hessian to test for convexity in optimization problems.

Derive the gradient descent update rule and analyze its convergence properties for smooth convex functions.

Derive the momentum and Nesterov accelerated gradient updates and explain their effect on navigating ill-conditioned landscapes.

Derive the Adam optimizer from bias-corrected first and second moment estimates of the gradients.

Analyze the role of learning rate schedules (e.g., warmup, cosine decay) in training dynamics.

Describe the challenges posed by saddle points in high-dimensional non-convex optimization and how stochasticity helps escape them.