Neural Network Foundations

Derive the backpropagation algorithm for a simple, fully-connected neural network.

Implement forward and backward propagation from scratch for a multi-layer perceptron.

Explain the role of various activation functions (e.g., ReLU, Sigmoid) and weight initialization strategies.

Apply batch gradient descent and stochastic gradient descent (SGD) and compare their training dynamics.

Implement momentum-based optimization and explain how it helps accelerate SGD convergence.

Implement the Adam optimizer and explain its adaptive learning rate mechanism.

Implement dropout regularization to prevent overfitting in a neural network.

Implement batch normalization and explain its effect on stabilizing the training process.