Classic Machine Learning

Derive the maximum likelihood estimation (MLE) framework for parameter estimation.

Apply Bayes' theorem to formulate posterior distributions for classification and parameter estimation.

Derive maximum a posteriori (MAP) estimation and connect it to regularization through the choice of prior.

Derive the density function of the multivariate Gaussian distribution and compute its conditional and marginal distributions.

Define Kullback-Leibler (KL) divergence and explain its relationship to cross-entropy and log-likelihood maximization.

Derive the closed-form solution (normal equations) for ordinary least squares (OLS) regression.

Interpret OLS as maximum likelihood estimation under a Gaussian noise assumption.

Explain the geometric interpretation of OLS as an orthogonal projection and analyze how multicollinearity affects the solution's stability.

Derive weighted least squares regression and explain its application for heteroscedastic data.

Use polynomial and radial basis functions to fit non-linear data within the linear regression framework.

Implement OLS regression from scratch in NumPy, including basis function expansion.

Derive Ridge regression (L2) as MAP estimation with a Gaussian prior and interpret its geometric effect on the coefficients.

Derive the Lasso (L1) regression objective and explain its sparsity-inducing property.

Derive the bias-variance decomposition of the mean squared error.

Analyze learning curves to diagnose underfitting and overfitting, and explain how regularization strength controls the bias-variance tradeoff.

Define and compare Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) for model selection.

Implement k-fold cross-validation from scratch to select hyperparameters like regularization strength.

Derive the Perceptron algorithm and state its convergence theorem for linearly separable data.

Derive the logistic regression model from the Bernoulli distribution and the sigmoid link function.

Derive the cross-entropy loss as the negative log-likelihood for logistic regression and prove its convexity.

Derive the gradient update rules for logistic regression parameters.

Extend logistic regression to multiclass classification by deriving the softmax regression formulation.

Implement logistic regression from scratch in NumPy, including support for multiclass classification.

Derive the Gaussian Discriminant Analysis (GDA) model and its maximum likelihood parameter estimates.

Show that GDA with a shared covariance matrix results in a linear decision boundary (Linear Discriminant Analysis).

Derive the Naive Bayes classifier from the conditional independence assumption.

Apply Laplace smoothing to handle zero-frequency problems in Naive Bayes and justify it as MAP estimation.

Compare generative vs. discriminative classifiers (e.g., Naive Bayes vs. Logistic Regression) in terms of asymptotic performance and data efficiency.

Implement a Gaussian Naive Bayes classifier from scratch in NumPy.

Formulate the hard-margin Support Vector Machine (SVM) as a constrained optimization problem and interpret it geometrically.

Derive the Lagrangian dual formulation of the hard-margin SVM, introducing the concept of support vectors.

Extend the formulation to the soft-margin SVM using slack variables and explain the role of the regularization parameter C.

Derive the kernel trick and explain how it enables non-linear classification without explicit high-dimensional feature mapping.

Describe common kernel functions (e.g., Polynomial, Radial Basis Function) and their properties.

Compare SVMs with logistic regression in terms of their loss functions, decision boundaries, and sensitivity to outliers.

Implement a simple soft-margin SVM solver using a quadratic programming library.

Describe the K-Nearest Neighbors (KNN) algorithm and analyze its computational complexity.

Explain the 'curse of dimensionality' and its impact on distance-based methods like KNN.

Derive information gain using entropy and compare it with Gini impurity as splitting criteria for decision trees.

Trace the CART algorithm to construct a classification tree, including finding the optimal split.

Analyze pruning strategies (pre-pruning and post-pruning) in terms of their effect on the bias-variance tradeoff.

Implement a CART-style decision tree classifier from scratch, including split selection and recursive building.

Explain how bootstrap aggregating (bagging) reduces variance and how Random Forests extend bagging to de-correlate trees.

Derive the AdaBoost algorithm's weight update rules from the perspective of minimizing an exponential loss function.

Formulate Gradient Boosting as a form of functional gradient descent.

Compare the mechanisms of Bagging, Random Forests, and Boosting in terms of their impact on bias and variance.

Implement the core components of a Random Forest classifier (bagging and feature subsampling) using your previously built decision tree.

Define the Probably Approximately Correct (PAC) learning framework and what it means for a hypothesis class to be efficiently learnable.

Derive the sample complexity bound for a finite hypothesis class.

Define VC dimension and compute it for linear classifiers in a 2D plane.

Explain the principle of Structural Risk Minimization and its connection to regularization and model selection.

State the 'No Free Lunch' theorem and articulate its implications for algorithm selection and the myth of a single 'best' algorithm.

Derive the K-means clustering algorithm as coordinate descent on an objective function.

Introduce the concept of latent variables and formulate the Gaussian Mixture Model (GMM).

Derive the general Expectation-Maximization (EM) algorithm as a method for finding MLE in models with latent variables.

Apply the EM algorithm to derive the update rules for fitting a GMM.

Explain the connection between K-means and EM for GMMs as a hard vs. soft assignment distinction.

Implement K-means clustering from scratch in NumPy.

Formulate the matrix factorization objective for collaborative filtering, including regularization terms.

Derive the update rules for matrix factorization using Alternating Least Squares (ALS).

Represent conditional independence structures using directed graphical models (Bayesian Networks).

Apply the d-separation criterion to determine conditional independence relationships in a given Bayesian Network.

Define a Hidden Markov Model (HMM) by its parameters (transition, emission, initial probabilities).

Derive the Forward Algorithm for computing the likelihood of an observation sequence (Evaluation).

Derive the Viterbi Algorithm for finding the most likely sequence of hidden states (Decoding).

Explain how the Baum-Welch algorithm uses the EM framework (with forward-backward results) to learn HMM parameters (Learning).

Implement the Viterbi algorithm from scratch in NumPy.