Hello! Welcome back to our module on Neural Network Foundations.
In our last lesson, we established a crucial principle: a neural network builds its predictive power by composing linear transformations with non-linear activation functions. We saw that without this non-linearity, even a very deep network would be no more powerful than a simple linear model.
Today, we will build directly on that foundation to address our next learning outcome: to explain the purpose of common activation functions like ReLU and Sigmoid. You've learned that we need non-linear activations; now you'll learn which ones to use, where to use them, and most importantly, why. Understanding this is a key step toward your goal of being able to justify your modeling decisions.
1. Why We Need Different Activation Functions
In the previous lesson, we treated the activation function as a single concept. In practice, the choice of activation function is a critical design decision, and we often use different ones in different parts of the network. The two main locations are:
- Hidden Layers: These are the intermediate layers of the network. Their primary job is to transform the data in a way that makes the final task easier. Here, the choice of activation function heavily influences how well the network can learn and the efficiency of the training process.
- Output Layer: This is the final layer that produces the model's prediction. The activation function here is chosen to format the output to match the problem you are solving (e.g., a probability for classification, a continuous value for regression).
Let's begin by deepening our understanding of the role of activation functions.
Activation Function in Neural Networks: Sigmoid, Tanh ...
The article 'Activation Function in Neural Networks' from Medium provides an excellent overview of the importance of activation functions. It uses a helpful analogy and highlights their key roles beyond just adding non-linearity.
Please read the introduction and the section titled 'The Importance of Activation Functions'. Pay close attention to the four points it raises, especially Gradient Propagation. This concept is central to today's lesson.
As the article emphasizes, the derivative of the activation function is critical for gradient propagation during training. This is a concept we'll return to as we examine our first function, Sigmoid.
2. The Sigmoid Function: For Probabilistic Outputs
The Sigmoid function was historically very popular and is still essential for specific tasks. It is defined by the following formula:
Its most important characteristic is that it takes any real number as input and "squashes" it into a value between 0 and 1. This makes it ideal for binary classification problems.
Purpose and Use Case
Because its output range is (0, 1), the Sigmoid function is the perfect choice for the output layer of a binary classification network. The output can be directly interpreted as the predicted probability that the input belongs to the positive class. For example, in a model that predicts if an email is spam, an output of 0.95 from a Sigmoid function means the model is 95% confident that the email is spam.
Let's watch a short video that introduces the Sigmoid function and its primary use.
Activation Functions In Neural Networks Explained | Deep Learning Tutorial
This video from AssemblyAI provides a concise explanation of the Sigmoid function and where it's typically used.
Watch from 02:26 to 03:10. The video clearly shows the function's graph and highlights its use in the final layer for binary classification.
The Vanishing Gradient Problem
While Sigmoid is great for output layers in binary classification, it has fallen out of favor for hidden layers. The reason is a major issue known as the vanishing gradient problem.
Look at the S-shape of the Sigmoid function again. For very large positive or very large negative inputs, the curve becomes almost completely flat. In calculus terms, this means its derivative (or slope) is very close to zero.
As we'll see in the next lesson, neural networks learn by passing gradients (error signals) backward through the network. If an activation function has a tiny gradient, it effectively shrinks the error signal passing through it. In a deep network with many layers, these signals can shrink so much that by the time they reach the early layers, they are practically zero. When the gradient vanishes, the weights of those early layers stop updating, and the network stops learning.
The following video provides an excellent visual and conceptual explanation of this problem.
Activation Functions - EXPLAINED!
The CodeEmporium channel explains the vanishing gradient problem in a very intuitive way, showing how the 'squeezing' nature of the Sigmoid function causes learning to stall.
Watch the two segments from 01:52-02:52 and 07:08-08:29. The first part introduces the problem, and the second part gives a more detailed look at why the Sigmoid's derivative leads to this issue. Focus on the intuition that the flat ends of the function have near-zero slope, which kills the gradient.
Test your understanding!
You are building a deep neural network (e.g., 20 layers) for a complex task. A colleague suggests using the Sigmoid activation function for all the hidden layers because its math is well-understood. Based on what you've just learned, what is the primary risk of this design choice, and how would it likely affect the training process?
Show answer
The primary risk is the vanishing gradient problem. Because the Sigmoid function's derivative is very small for inputs that aren't close to zero, the gradients would likely shrink exponentially as they are propagated backward through the 20 layers. This would cause the early layers of the network to learn extremely slowly, or not at all, as their weight updates would be close to zero. The training process would likely stall, and the model would fail to converge to a good solution.
3. Rectified Linear Unit (ReLU): The Modern Default
To solve the vanishing gradient problem, researchers proposed a much simpler but more effective activation function: the Rectified Linear Unit, or ReLU.
Its formula is simply:
This means if the input is positive, the output is the input itself. If the input is negative, the output is zero.
Purpose and Use Case
ReLU has become the standard and default choice for hidden layers in almost all modern neural networks, from simple feed-forward networks to complex Convolutional Neural Networks (CNNs).
Here are the key advantages of ReLU over Sigmoid for hidden layers:
- Solves Vanishing Gradients: For any positive input, the derivative of ReLU is 1. This means the gradient can be passed backward through the network without shrinking, allowing deep networks to learn effectively.
- Computational Efficiency: The
max(0, z)operation is trivial for a computer to perform, unlike the exponentiale^zin the Sigmoid function. This makes training significantly faster. - Sparsity: Because all negative inputs are mapped to zero, a portion of the neurons in the network will output zero at any given time. This "sparsity" can make the network more efficient and can lead to more robust and disentangled representations of the data.
Let's watch a brief overview of ReLU.
Activation Functions In Neural Networks Explained | Deep Learning Tutorial
The same video from AssemblyAI also gives a great introduction to the ReLU function.
Watch from 03:33 to 04:05. This segment introduces ReLU and its simple but powerful logic.
The Dying ReLU Problem
ReLU is not without its own potential issue, known as the dying ReLU problem.
If a neuron's weights are updated in such a way that the input to its ReLU activation is always negative, the neuron will always output zero. Consequently, the gradient flowing through that neuron will also always be zero. The neuron gets "stuck" in an off state and can no longer participate in learning. It effectively becomes "dead".
This is less common in practice than the vanishing gradient problem with Sigmoid, but it's something to be aware of. To combat this, variants like Leaky ReLU were introduced, which allow a small, non-zero gradient for negative inputs.
Activation Functions - EXPLAINED!
The CodeEmporium video also provides a great explanation for why the dying ReLU problem occurs.
Now watch the segments from 02:31-03:48 and 08:29-09:24. They first introduce ReLU as a solution to vanishing gradients and then explain how it can sometimes lead to 'dead' neurons.
4. Practical Guidelines: Putting It All Together
So, how do you choose? Fortunately, there are some well-established rules of thumb that will serve you well in most situations.
Introduction to Activation Functions in Neural Networks
The DataCamp article 'Introduction to Activation Functions in Neural Networks' provides clear, actionable advice on this topic.
Read the short section 'Choosing the Right Activation Function'. It provides a simple decision framework that summarizes our discussion.
To summarize the common practice:
- For Hidden Layers: Start with ReLU. It is the most common and effective choice for hidden layers. If you notice that your model's training is stalling unexpectedly, you might investigate a variant like Leaky ReLU, but ReLU is the go-to default.
- For the Output Layer: The choice depends entirely on your problem.
- Binary Classification: Use Sigmoid to get a single probability output between 0 and 1.
- Multi-Class Classification: Use Softmax (which we will cover in a later module). It's a generalization of Sigmoid for more than two classes.
- Regression (predicting a continuous value): Use no activation function (or a "linear" activation). This allows the output to be any real number.
Test your understanding!
You are designing a neural network to predict house prices (a regression task) based on features like square footage and number of bedrooms. What activation functions would you choose for the hidden layers and the output layer, and why?
Show answer
- Hidden Layers: I would use ReLU. It's computationally efficient and avoids the vanishing gradient problem, allowing the network to effectively learn the complex, non-linear relationships between house features and price.
- Output Layer: I would use no activation function (linear activation). House prices are continuous real numbers, not probabilities. A linear output allows the model to predict any value (e.g., $350,000, $1,200,000), which matches the nature of the regression problem. Using Sigmoid or ReLU here would improperly constrain the output.
Conclusion
In this lesson, we moved from the general need for non-linearity to the specific purposes of two of the most important activation functions in deep learning.
Key Takeaways:
- The choice of activation function is a critical design decision that depends on the layer's position (hidden vs. output) and the overall task.
- Sigmoid squashes inputs to a (0, 1) range, making it ideal for the output layer in binary classification where a probability is needed. However, it suffers from the vanishing gradient problem in deep hidden layers.
- ReLU is the modern default for hidden layers. It is computationally fast and prevents vanishing gradients, but it can sometimes suffer from the dying ReLU problem.
- The general rule is: ReLU for hidden layers, and an activation tailored to the task for the output layer.
Preview of the Next Lesson:
We now have all the architectural components of a simple neural network: linear layers and non-linear activations. But how does the network actually learn the right weights and biases to make accurate predictions? In the next lesson, we will dive into the training process itself, exploring how gradient descent and backpropagation work together to optimize a model. Your understanding of gradients from today's lesson will be a great stepping stone