Multi-Layer Feedforward Neural Network Architecture

1. The Big Picture: A Network of Neurons

At its core, an Artificial Neural Network (ANN) is a computational model inspired by the structure and function of the human brain. It consists of interconnected nodes, or "neurons," organized in layers.

To build a strong mental model, let's start with a fantastic visual explanation from the 3Blue1Brown channel.

As you saw, the network is composed of layers. The first layer holds the input data (pixel values), the last layer holds the final output (the network's confidence for each digit), and the "hidden" layers in between are where the magic happens. Now, let's zoom in and see what each individual neuron is actually doing.

2. The Building Block: The Artificial Neuron

A single artificial neuron is a simple computational unit. It takes in several inputs, processes them, and produces a single output. This process always involves three steps:

1. Compute a Weighted Sum: Each input connection to the neuron has a weight. The neuron multiplies each input value by its corresponding weight and sums them all up. A higher weight means the neuron considers that input more important.
2. Add a Bias: After the weighted sum is calculated, a bias value is added. You can think of the bias as a way to make the neuron more or less likely to activate, independent of its inputs. It's an adjustable "knob" that helps the neuron fit the data better.
3. Apply an Activation Function: The result (weighted sum + bias) is passed through an activation function. This function "squashes" the value into a desired range (e.g., between 0 and 1) and introduces non-linearity, which is crucial for the network's power.

This next clip from the same 3Blue1Brown video animates this process beautifully.

So, a neuron's job is to "light up" (have a high activation) when it recognizes a specific pattern in its inputs. The weights define what pattern it's looking for, and the bias determines how strong that pattern needs to be before the neuron starts activating meaningfully.

3. Assembling the Network: The Multi-Layer Structure

Now, we arrange these simple neurons into a Multi-Layer Feedforward Network, also known as a Multilayer Perceptron (MLP). The structure consists of three types of layers.

Let's formally define each part.

• Input Layer: This is the entry point of the network. It doesn't contain processing neurons; it simply holds the initial data for your problem. The number of nodes in the input layer is equal to the number of features in your dataset (e.g., for a 28x28 pixel image, you'd have 784 input nodes).

• Hidden Layers: These are the layers between the input and output. A network can have one or more hidden layers. This is where the bulk of the computation happens. Each neuron in a hidden layer receives inputs from the previous layer, performs its calculation (weighted sum + bias + activation), and passes its output to the next layer. These layers allow the network to learn complex patterns and relationships in the data that a simple model couldn't capture.

• Output Layer: This is the final layer of the network. It produces the ultimate result of the computation. The structure of the output layer depends on the problem you're solving:

  • For regression (predicting a value like a price), it might be a single neuron with a linear activation function.
  • For binary classification (predicting one of two classes), it's often a single neuron with a sigmoid activation function (outputting a probability between 0 and 1).
  • For multi-class classification (predicting one of many classes), it's typically one neuron per class, using a softmax activation function.

The term feedforward means that information flows in only one direction—from the input layer, through the hidden layers, to the output layer. There are no loops or backward connections in the network's structure.

To solidify these definitions, let's consult a clear, concise written resource.

The Crucial Role of Non-Linearity

You might wonder why we bother with activation functions like sigmoid. Why not just pass the weighted sum directly to the next layer? This is a critical point. If we only used linear operations (like sums and multiplications), a deep stack of layers would be mathematically equivalent to a single, simple linear layer. The non-linear activation functions are what allow the network to learn far more complex, curved decision boundaries and approximate any continuous function.

This short video clip explains this concept perfectly.

4. A Compact View: The Power of Linear Algebra

As you can imagine, writing out the calculations for every single neuron connection would be incredibly tedious. This is where your knowledge from the linear algebra module comes in handy. The entire forward pass for a layer can be expressed with a single, elegant matrix equation.

Let:

• $a^{(l-1)}$ be the vector of activations from the previous layer ($l-1$).
• $W^{(l)}$ be the matrix of weights for the current layer ($l$), where each row corresponds to the weights for one neuron.
• $b^{(l)}$ be the vector of biases for the current layer ($l$).
• $\sigma$ be the activation function.

Then, the vector of activations for the current layer, $a^{(l)}$, is:

This compact form is not just for notation; it's how libraries like PyTorch and TensorFlow perform these calculations at incredible speeds, leveraging highly optimized matrix operations on GPUs.

Let's watch one last clip that visualizes this matrix representation.

Conclusion

Great work today! You've taken a significant first step into understanding the architecture of neural networks. These concepts are the foundation upon which all of deep learning is built.

Key Takeaways:

• A multi-layer feedforward neural network is composed of an input layer, one or more hidden layers, and an output layer.
• The fundamental unit is the neuron, which computes a weighted sum of its inputs, adds a bias, and applies a non-linear activation function.
• Information flows in one direction (feedforward) from input to output.
• The entire structure—the number of layers and the number of neurons per layer—is called the network's architecture.
• Non-linear activation functions are essential for the network to learn complex patterns.
• Linear algebra provides a compact and efficient way to represent and compute the network's operations.

Preview of the Next Lesson:
Today, we learned that neurons use activation functions, but we only briefly mentioned a few like sigmoid. In the next lesson, we will dive deeper and explain the role of different activation functions like Sigmoid, ReLU, and Softmax. You will learn what each one does, where it's typically used, and why modern networks often prefer one over the other.