Matrix Decompositions for Model Analysis

Derive the singular value decomposition (SVD) and relate singular values to the eigenvalues of AᵀA.

Apply SVD to analyze matrix properties like rank, condition number, and the geometry of the four fundamental subspaces.

Utilize truncated SVD for low-rank approximation and connect it to neural network compression techniques like LoRA.

Interpret eigendecomposition geometrically and connect Hessian eigenvalues to the curvature of the loss landscape.

Apply the spectral theorem to symmetric matrices arising as covariance matrices, kernel matrices, and the Fisher information matrix.

Derive Principal Component Analysis (PCA) from the spectral theorem on the covariance matrix and interpret components as variance-maximizing directions.