Understanding Activation Functions

The Critical Need for Non-Linearity

In the previous lesson, we stated that activation functions introduce non-linearity. But why is this so important? What would happen if we skipped the activation function, or used a simple linear one like $f(z)=z$, also called the identity function?

A network built this way would only be capable of learning linear relationships. No matter how many layers of neurons you stack, the entire network would behave as if it were just a single, simple linear model.

To see the mathematical proof of this, let's turn to a video by Andrew Ng, a leading figure in AI education.

As you saw, if we have:

• Layer 1: $a_1=W_{1}x+b_1$ (linear activation)
• Layer 2: $a_2=W_2{a}_1+b_2$ (linear activation)

By substituting the first equation into the second, we get:
$a_2=W_2(W_{1}x+b_1)+b_2$
$a_2=(W_2{W}_1)x+(W_2{b}_1+b_2)$

If we define $W^{\prime }=W_2{W}_1$ and $b^{\prime }=W_2{b}_1+b_2$, the equation becomes:
$a_2=W^{\prime }x+b^{\prime }$

This is just another linear function. Adding more hidden layers would just result in a more complex, but still fundamentally linear, transformation. Since most real-world problems are highly non-linear, such a network would be severely limited.

The only common exception, as the video also mentions, is using a linear activation function in the output layer for regression tasks, where the goal is to predict a continuous value (like a stock price or temperature) that is not constrained to a specific range.

A Tour of Common Activation Functions

Now that we've established the need for non-linearity, let's explore the functions that provide it. Each has a unique shape and properties that make it suitable for different parts of a neural network.

This next video provides an excellent visual survey of the most popular activation functions.

Let's break these down in more detail, adding their mathematical formulas and discussing their pros and cons.

1. Sigmoid (or Logistic) Function

The Sigmoid function was a popular choice in the early days of neural networks.

Formula:
$\sigma (z)=\frac{1}{1+e^{-z}}$

Properties:

• Output Range: (0, 1). It squashes any real number into this range.
• Shape: A smooth 'S'-shaped curve.
• Use Case: Primarily used in the output layer for binary classification. The output can be interpreted as the probability of the positive class.

Limitations:

• Vanishing Gradients: For very large positive or negative inputs, the function's slope (derivative) becomes extremely close to zero. During backpropagation, these tiny gradients can make learning incredibly slow or halt it altogether, especially in deep networks.
• Not Zero-Centered: The output is always positive. This can lead to less efficient updates during gradient descent.

2. Tanh (Hyperbolic Tangent)

The Tanh function is a close cousin of the Sigmoid function.

Formula:
$\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$

Properties:

• Output Range: (-1, 1).
• Shape: Also 'S'-shaped, but steeper than Sigmoid.
• Advantage over Sigmoid: Its output is zero-centered. This property often helps the network learn more efficiently than with Sigmoid. For this reason, Tanh is generally preferred over Sigmoid for hidden layers.

Limitations:

• Vanishing Gradients: Like Sigmoid, Tanh saturates at its extremes, leading to the same vanishing gradient problem.

3. ReLU (Rectified Linear Unit)

ReLU has become the default and most popular activation function for hidden layers in modern neural networks.

Formula:
$\text{ReLU}(z)=\max (0,z)$

Properties:

• Output Range: [0, ∞).
• Shape: It's a simple function: 0 for all negative inputs, and a straight line $f(z)=z$ for all positive inputs.
• Use Case: The standard choice for hidden layers.

Advantages:

• Computationally Efficient: Very simple and fast to compute.
• Avoids Vanishing Gradients (for positive inputs): The gradient is a constant 1 for positive inputs, allowing gradients to flow well during backpropagation.
• Sparsity: Because it outputs 0 for half of its input range, it tends to make some neurons "inactive," which can make the network more efficient.

Limitations:

• The "Dying ReLU" Problem: If a neuron's weights are updated in such a way that its input $z$ is always negative, it will always output 0. Consequently, the gradient flowing through it will also be 0, and it will never update its weights again. The neuron effectively "dies" and stops contributing to learning.

To address this, variants like Leaky ReLU were introduced. Leaky ReLU has the formula $f(z)=\max (\alpha z,z)$, where $\alpha$ is a small constant like 0.01. This gives a small, non-zero slope for negative inputs, preventing the neuron from dying.

4. Softmax Function

Softmax is unique because it acts on an entire vector of outputs from the final layer, not just a single neuron's output.

Formula: For a vector of raw outputs (logits) $Z=[z_1,z_2,...,z_k]$, the Softmax function is applied to each element $z_i$:
$\text{Softmax}(z_i)=\frac{e^{z_i}}{{\sum }_{j=1}^k{e}^{z_j}}$

Properties:

• Output: It converts a vector of arbitrary real numbers into a probability distribution. Each output value is in the range (0, 1), and all output values sum to 1.
• Use Case: Used exclusively in the output layer for multi-class classification. Each output corresponds to the probability of the input belonging to a specific class.

For a deeper dive into these functions, including their derivatives and more detailed pros and cons, the following article is an excellent resource.

How to Choose the Right Activation Function

You now have a toolbox of activation functions. The final step is learning some practical rules for selecting the right tool for the job.

Here are the most common rules of thumb, which are also summarized well in the V7 Labs article:

1. For Hidden Layers:

   • Start with ReLU. It's the most common and often the best-performing choice.
   • If you notice your model is not training well due to many "dead" neurons, switch to a variant like Leaky ReLU.
   • Tanh is a viable alternative, but Sigmoid should generally be avoided in hidden layers.

2. For the Output Layer:

   • Regression (predicting a continuous value): Use a Linear activation function (or no activation).
   • Binary Classification (predicting one of two classes): Use the Sigmoid function.
   • Multi-class Classification (predicting one of many classes): Use the Softmax function.

Conclusion

In this lesson, we explored the indispensable role of activation functions in neural networks. You learned that they are the source of the non-linearity that allows networks to model the complex patterns found in real-world data.

Key Takeaways:

• Without non-linear activation functions, even a deep neural network collapses into a simple linear model.
• ReLU is the modern standard for hidden layers due to its efficiency and ability to mitigate vanishing gradients, though it can suffer from the "Dying ReLU" problem.
• Sigmoid is used in the output layer for binary classification, converting a raw score into a probability between 0 and 1.
• Softmax is used in the output layer for multi-class classification, converting a vector of scores into a probability distribution where all class probabilities sum to 1.
• The choice of activation function for the output layer is determined by the problem type (regression, binary classification, or multi-class classification).

Preview of the Next Lesson:
We have now examined the neuron's internal components in detail. The next logical step is to see how these neurons are assembled. In the upcoming lesson, we will describe the architecture of a multi-layer feedforward neural network, learning how layers of neurons are connected to process information from input to output.