Understanding Activation Functions

1. Why Activation Functions are Essential (A Quick Recap)

In the previous lesson, you saw that without a non-linear activation function, a deep neural network with many layers would be mathematically equivalent to a simple, single-layer linear model. This would severely limit its power, making it incapable of learning the complex, non-linear relationships present in most real-world data.

To reinforce why introducing non-linearity is so critical, let's watch a short, clear explanation.

Now that we've solidified why we need them, let's explore the most common ones.

2. A Gallery of Activation Functions

There are many activation functions, but a few have become standard choices in deep learning. We'll focus on the three most important ones for your GATE syllabus: Sigmoid, ReLU, and Softmax.

The image below shows the shapes and formulas for some of these key functions. Keep these graphs in mind as we discuss each one.

3. Activations for Hidden Layers

The choice of activation function in the hidden layers dramatically affects how well and how fast a network can be trained.

The Sigmoid Function

The Sigmoid function was historically very popular. It takes any real-valued number and "squashes" it into a range between 0 and 1.

• Formula: $\sigma (x)=\frac{1}{1+e^{-x}}$
• Output Range: (0, 1)

Its S-shaped curve means it's sensitive to changes around an input of 0 but becomes flat for very large positive or negative inputs. While its output range is useful for representing probabilities (as we'll see later), it has a significant drawback when used in hidden layers.

To understand this drawback, let's turn to a detailed written explanation.

The vanishing gradient problem you just read about is a major reason why Sigmoid is no longer the default choice for hidden layers in deep networks. When the function's output is close to 0 or 1, its gradient is almost zero. During backpropagation, these tiny gradients get multiplied through many layers, causing the updates to the weights in the earlier layers to become so small ("vanish") that the network effectively stops learning.

The Rectified Linear Unit (ReLU)

To combat the vanishing gradient problem, the Rectified Linear Unit (ReLU) was introduced and is now the most popular activation function for hidden layers. Its definition is surprisingly simple.

• Formula: $f(x)=\max (0,x)$
• Output Range: [0, ∞)

In short, if the input is negative, the output is 0. If the input is positive, the output is the input itself.

Let's watch a brief video segment that explains ReLU.

Advantages of ReLU:

• Computationally Efficient: The max(0, x) operation is much faster to compute than the exponential in the Sigmoid function.
• No Vanishing Gradient (for positive inputs): For positive inputs, the gradient is a constant 1, which allows learning to proceed without the gradients shrinking to zero.

However, ReLU has its own potential issue, known as the "Dying ReLU" problem. When a neuron's input is consistently negative, its output will always be zero. Consequently, the gradient flowing through it will also always be zero, and its weights will never be updated. The neuron effectively "dies" and stops participating in learning.

4. Activations for the Output Layer

The choice of activation function for the final output layer is determined by the specific task you are trying to solve.

Sigmoid for Binary Classification

While Sigmoid has fallen out of favor for hidden layers, it is still the standard choice for the output layer in a binary classification problem (e.g., cat vs. dog, spam vs. not spam).

Because its output is a single value between 0 and 1, it can be directly interpreted as the probability of the positive class. For example, an output of 0.92 could mean the network is 92% confident that the input image is a cat.

Softmax for Multi-Class Classification

What if you have more than two classes? For example, classifying handwritten digits (0 through 9). You need the network to output a probability for each class, and all these probabilities should sum to 1. This is the job of the Softmax function.

Softmax takes a vector of raw scores (called "logits") from the final layer and transforms them into a probability distribution.

• Formula: $S(z_i)=\frac{e^{z_i}}{{\sum }_{j=1}^K{e}^{z_j}}$ for class $i$ out of $K$ classes.

This might look complex, but the idea is simple:

1. Take the exponential of each raw score to make them all positive.
2. Divide each exponentiated score by the sum of all exponentiated scores to normalize them, ensuring they all add up to 1.

Let's watch a detailed walkthrough of this process.

The diagram below shows how the Softmax function operates as the final step in a network layer for multi-class classification.

Linear for Regression

For regression tasks, where you are predicting a continuous value (e.g., the price of a house, the temperature tomorrow), you often use a linear activation function ($f(x)=x$) or no activation function at all in the output layer. This allows the network to output any real number, not just a value constrained between 0 and 1.

5. Summary: How to Choose an Activation Function

You now have a solid understanding of the key activation functions. Here are some simple rules of thumb for choosing them, which will be very useful for your GATE preparation and future projects.

Conclusion

Great job! You've now demystified one of the most important components of a neural network. Understanding not just what these functions are, but why and where to use them, is a critical skill for any AI practitioner.

Key Takeaways:

• Activation functions introduce the non-linearity that allows deep networks to learn complex patterns.
• ReLU ($f(x)=\max (0,x)$) is the standard, computationally efficient choice for hidden layers, though it can suffer from the "dying ReLU" problem.
• Sigmoid ($\sigma (x)=\frac{1}{1+e^{-x}}$) squashes outputs to a (0, 1) range. It's used for binary classification output layers but is generally avoided in hidden layers due to the vanishing gradient problem.
• Softmax is used for multi-class classification output layers. It converts a vector of raw scores into a probability distribution where all outputs sum to 1.
• For regression problems, the output layer typically uses a linear activation function.

Preview of the Next Lesson:
In our last lesson, you learned about the structure of a network. Today, you learned about the activation functions inside it. In our next lesson, "Implement the full forward pass of a feedforward neural network to compute predictions and loss," we will bring all of this together. You will see how to combine these concepts in code to take an input and make it flow through the entire network to produce a final prediction.