Vector Calculus for Automatic Differentiation

Compute gradients of scalar-valued functions of vectors and matrices using numerator and denominator layout conventions.

Compute Jacobians of vector-valued functions and apply the chain rule to compositions of neural network layers.

Derive the gradients of common loss functions (MSE, cross-entropy) with respect to network logits.

Derive the Jacobian of the softmax function and its simplification when composed with the cross-entropy loss.

Apply matrix calculus identities (e.g., for d(AX), d(XᵀAX), d(log det X)) used in VAEs and Normalizing Flows.

Trace a full backpropagation pass through a small MLP by hand using Jacobian-vector products (JvPs).

Compute Hessians and explain the rationale behind the Gauss-Newton and Fisher Information Matrix approximations.