Understanding Feed-Forward Neural Networks (MLPs)

1. The Basic Architecture: Layers of Neurons

At its core, an Artificial Neural Network (ANN) is a computational model inspired by the structure of the human brain. It's composed of interconnected nodes, or neurons, organized into layers. For the type of network we're studying today, a Multi-Layer Perceptron (MLP), these layers are:

• Input Layer: This is where the network receives data. Each neuron in this layer corresponds to a single feature of your input data (e.g., a pixel in an image, a column in a spreadsheet).
• Hidden Layers: These are the intermediate layers between the input and output. An MLP must have at least one hidden layer. This is where the majority of the computation happens, as the network learns to identify patterns and relationships in the data.
• Output Layer: This layer produces the final result. The structure of this layer depends on the task:
  • For regression, it might be a single neuron outputting a continuous value.
  • For binary classification, it might be a single neuron outputting a probability.
  • For multi-class classification, it might be multiple neurons, each representing the probability of a specific class.

Information in an MLP flows in one direction—from the input layer, through the hidden layers, to the output layer. This is why it's called a feed-forward network.

Here is a simple diagram illustrating this structure.

2. The Neuron: A Simple Computational Unit

To understand the network, we first need to understand its fundamental component: the neuron. Think of a single neuron as a simple decision-making unit.

The following short video provides a quick, high-level overview. The key idea to grasp is that each neuron functions like its own mini-model.

As the video explains, each neuron performs two main steps:

1. Calculate a Weighted Sum: It takes all the outputs from the neurons in the previous layer and multiplies each by a weight. These weights signify the importance of each input connection. It then sums these products and adds a bias. The bias is a trainable value that helps shift the neuron's activation threshold, giving the model more flexibility.
2. Apply an Activation Function: The resulting weighted sum is then passed through a non-linear activation function. This is a crucial step! Without a non-linear activation function, a deep neural network would just be equivalent to a single, complex linear model, no matter how many layers it had. The non-linearity is what allows the network to learn incredibly complex patterns.

3. Building Intuition: How Layers Work Together

Now, let's zoom out from a single neuron to the entire network. How do these simple units, arranged in layers, accomplish complex tasks like image recognition? The magic lies in how they learn hierarchical features.

The following video from 3Blue1Brown is one of the best explanations of a neural network's structure and intuition.

This idea of a hierarchy of features is fundamental to deep learning. Each layer takes the simple patterns found by the previous layer and combines them into something more abstract and complex.

4. The Components of an MLP: A Formal Summary

You now have the visual and conceptual intuition. To solidify your knowledge for the GATE exam, let's review the formal definitions of these components.

5. The Math and Modern Practices

As the resources have shown, the core computation in a neuron is a weighted sum plus a bias, followed by an activation function. For a neuron's output $a$, with inputs $x_i$, weights $w_i$, and bias $b$, this is:

where $f$ is the activation function.

Given your background in linear algebra, you'll recognize that the computation for an entire layer can be expressed very efficiently using matrix operations. If W is the matrix of weights, x is the vector of inputs from the previous layer, and b is the vector of biases, the vector of activations a for the current layer is:

The 3Blue1Brown video you watched earlier has an excellent segment explaining this compact notation.

The key takeaways from that final segment are:

• Sigmoid: An older activation function that squashes values to be between 0 and 1. It was inspired by the biological idea of a neuron "firing" or not.
• ReLU (Rectified Linear Unit): The modern default. Its function is simply $f(x)=\max (0,x)$. It's computationally cheaper and has been found to work better for training very deep networks.

Conclusion

In this lesson, we've unpacked the basic structure of a Multi-Layer Perceptron, the foundational model of deep learning. You've seen how it's built from simple units (neurons) arranged in layers, and how these layers work together to learn hierarchical representations of data.

Key Takeaways:

• An MLP is a feed-forward neural network with an input layer, one or more hidden layers, and an output layer.
• Each neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a non-linear activation function.
• Weights and biases are the trainable parameters of the network.
• Activation functions (like ReLU) introduce non-linearity, which is essential for learning complex patterns.
• Hidden layers build a hierarchy of features, with each layer learning more abstract concepts based on the outputs of the previous one.
• The entire network is a complex, but differentiable, function that maps inputs to outputs.

Preview of the Next Lesson:
This concludes our module on Supervised Learning. We will now shift gears to a different paradigm of machine learning. In the next module, Unsupervised Learning & Dimensionality Reduction, our first lesson will be: Explain partitioning clustering algorithms (k-means/k-medoids). We will explore how to find structure in data when you don't have any labels to guide you.