Activation Functions: Sigmoid, Tanh, and ReLU

Hello! Welcome back to our second module, Neural Networks from Scratch.

In our last session, we dissected the fundamental building block of any neural network: the single artificial neuron. You learned how it processes information by taking a weighted sum of inputs, adding a bias, and passing the result through an activation function. We used the sigmoid function as our example to compute a neuron's output.

Today, we'll answer a crucial question: why are there so many different activation functions? The choice of activation function is a key architectural decision that directly impacts a network's performance and training stability.

This lesson directly addresses the learning outcome: Describe and compare common activation functions (sigmoid, tanh, ReLU) and their properties. By the end of our 60 minutes, you will understand the characteristics, advantages, and disadvantages of these core functions and know when to use each one.

Why Do We Need Activation Functions?

As a quick refresher, the primary role of an activation function is to introduce non-linearity into the network. A neural network composed only of linear transformations (weighted sums) would, regardless of its depth, be equivalent to a single linear model. It would be incapable of learning the complex, non-linear patterns found in most real-world data.

The video below provides a concise explanation of why this non-linearity is so essential.

Now, let's dive into the most common functions you'll encounter.

1. The "Classic" Activation Functions: Sigmoid and Tanh

For a long time, sigmoid and tanh were the standard choices for neural networks. They are both S-shaped curves and are closely related.

The following video from Machine Learning Studio will be our guide. We'll watch specific segments for each function.

The Sigmoid Function

You're already familiar with sigmoid from our last lesson. It squashes any real-valued number into the range (0, 1), which makes it very useful for interpreting outputs as probabilities in binary classification.

Let's watch the detailed breakdown.

Key Properties of Sigmoid:

• Formula: $\sigma (x)=\frac{1}{1+e^{-x}}$
• Output Range: (0, 1)
• Advantage: Smooth, differentiable, and its output can be interpreted as a probability.
• Major Disadvantage: Vanishing Gradients. As you saw in the video, for very large positive or negative inputs, the sigmoid curve flattens, and its derivative approaches zero. During backpropagation (which we will cover in detail soon), these near-zero gradients get multiplied down through the layers, causing the weight updates for the initial layers to become infinitesimally small. This effectively "stalls" learning.
• Another Disadvantage: Not Zero-Centered. The output is always positive. This can lead to less efficient gradient updates during training.

The Tanh (Hyperbolic Tangent) Function

The Tanh function is a mathematically shifted version of the sigmoid function. It squashes values to a different range, which provides a significant advantage.

Key Properties of Tanh:

• Formula: $\tanh (x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$
• Output Range: (-1, 1)
• Advantage: Zero-Centered. Because the output is centered around zero, the gradients are not all pushed in the same direction. This generally helps the network learn more efficiently compared to using sigmoid in hidden layers.
• Disadvantage: It still suffers from the vanishing gradient problem in its saturated regions (the flat parts at either end), although its gradients are steeper than sigmoid's.

Because of its zero-centered nature, Tanh was generally preferred over Sigmoid for hidden layers in neural networks before the rise of our next function.

2. The Modern Standard: ReLU

The Rectified Linear Unit, or ReLU, has become the default activation function for hidden layers in most deep learning applications today. Its simplicity is its greatest strength.

Key Properties of ReLU:

• Formula: $f(x)=\max (0,x)$
• Output Range: [0, $\infty$)
• Advantages:
  • Computationally Efficient: It's a simple max operation, much faster than the exponentials in Sigmoid and Tanh.
  • Mitigates Vanishing Gradients: For positive inputs, the derivative is a constant 1. This means the gradient can flow backward through the network without shrinking, allowing deeper networks to train effectively.
• Major Disadvantage: The "Dying ReLU" Problem. As the video mentioned, if a neuron's weights are updated such that its input is always negative, it will always output 0. The gradient for negative inputs is also 0, so the neuron's weights will never be updated again. It becomes "stuck" and effectively "dies," playing no further part in the network.

A Simple Fix: Leaky ReLU

To address the dying ReLU problem, a small modification was proposed: the Leaky ReLU.

Leaky ReLU introduces a small, non-zero slope for negative inputs ($f(x)=\alpha x$ for $x<0$, where $\alpha$ is typically a small value like 0.01). This ensures the neuron never has a zero gradient and can always recover. It is often a good drop-in replacement for ReLU.

3. Comparison and Practical Guidance

Now that you've seen the individual functions, let's put it all together. The best way to compare them is to see their properties side-by-side.

The table you just reviewed is an excellent summary. Here is another version for a quick visual recap:

Rules of Thumb for Choosing Activation Functions:

Based on what you've learned, here are some practical guidelines used by AI practitioners:

1. For hidden layers, start with ReLU. It's the most common and generally most effective choice. If you encounter issues with dying neurons, switch to Leaky ReLU.
2. For the output layer:
   • Use Sigmoid for binary classification tasks, where you need a single probability output between 0 and 1.
   • Use a linear activation function (i.e., no activation) for regression tasks, where the output can be any real number.
   • For multi-class classification, you will use the Softmax function, which is a generalization of Sigmoid that produces a probability distribution across multiple classes. We will cover this in a future lesson.
3. Tanh is less common in modern architectures but can be a valid choice for hidden layers, especially if you have reasons to prefer a zero-centered output. You may also see it in older Recurrent Neural Network (RNN) designs.

Conclusion

In this lesson, you've moved beyond simply using an activation function to understanding the critical trade-offs between the most common options. This knowledge is fundamental to designing effective and stable neural networks.

Key Takeaways:

• Sigmoid: Outputs between (0, 1). Best for binary classification output layers but suffers from vanishing gradients and is not zero-centered.
• Tanh: Outputs between (-1, 1). Its zero-centered output is an improvement over Sigmoid for hidden layers, but it also suffers from vanishing gradients.
• ReLU: The modern default for hidden layers. It's computationally fast and avoids vanishing gradients for positive inputs but can suffer from the "dying ReLU" problem.
• Leaky ReLU: A simple fix for the dying ReLU problem that is often used as a more robust alternative.

Now that you understand both the structure of a single neuron and the activation functions that give it power, you are ready to start assembling them. In our next lesson, we will combine multiple neurons into layers and implement a complete forward pass for a multi-layer perceptron using only NumPy.