Activation Functions: Properties, Gradients, and Training Impact

Welcome back. In our previous lessons, we've defined a feedforward neural network as a composition of affine transformations and nonlinear activation functions: ${\mathbf{h}}^{(l)}=f({\mathbf{W}}^{(l)}{\mathbf{h}}^{(l-1)}+{\mathbf{b}}^{(l)})$. We also saw how to represent this entire computation as a graph, which sets the stage for the backpropagation algorithm.

Today, we will focus on that critical component: the nonlinear activation function, $f$. Its choice is not arbitrary; it profoundly influences a network's ability to learn and the dynamics of the training process. We will dissect four of the most important activation functions in the history and practice of deep learning: Sigmoid, Tanh, ReLU, and GELU. For each, we will analyze its mathematical definition, its graphical shape, and, most importantly, the behavior of its gradient—a property that has driven the evolution from one function to the next.

Your strong foundation in calculus will be invaluable here, as understanding the derivatives of these functions is the key to grasping concepts like vanishing gradients and their impact on training deep architectures.

The Classic Saturating Functions: Sigmoid and Tanh

Historically, the first widely used activation functions were the Sigmoid and Hyperbolic Tangent (Tanh). They were popular partly because their bounded, "S"-shaped curves were required by early versions of the Universal Approximation Theorem, which we will discuss in our next lesson.

Let's look at their familiar shapes and formulas.

While these functions are smooth and differentiable, they share a critical flaw that hindered the development of deep neural networks. The following video explains their properties and, crucially, their drawbacks.

To recap the key issues:

• Sigmoid ($\sigma (x)=\frac{1}{1+e^{-x}}$):

  • Pros: Squashes outputs to a (0, 1) range, which is interpretable as a probability and useful in the final layer of a binary classifier. It has a direct connection to the logistic regression model you're familiar with.
  • Cons:
    1. Vanishing Gradients: Its derivative, ${\sigma }^{\prime }(x)=\sigma (x)(1-\sigma (x))$, has a maximum value of 0.25 at $x=0$ and rapidly approaches zero elsewhere. In a deep network, these small gradients are multiplied together many times during backpropagation, causing the overall gradient to "vanish" and preventing weights in early layers from being updated effectively.
    2. Not Zero-Centered: Outputs are always positive. As the video explained, this forces all weight gradients in the next layer to have the same sign, leading to inefficient updates.

• Tanh ($\tanh (x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$):

  • Pros: Its output range of (-1, 1) is zero-centered, which resolves the inefficient update problem seen in Sigmoid. For this reason, Tanh is almost always preferred over Sigmoid for hidden layers.
  • Cons: It still saturates for large $|x|$, and its derivative, ${\tanh }^{\prime }(x)=1-{\tanh }^2(x)$, also goes to zero in these regions, leading to the same vanishing gradient problem.

A Breakthrough: The Rectified Linear Unit (ReLU)

The vanishing gradient problem was a major barrier to training deep networks. The solution turned out to be surprisingly simple: the Rectified Linear Unit, or ReLU.

ReLU($x$) = max(0, $x$)

This function was a pivotal innovation that made training much deeper networks feasible.

The success of ReLU led to several variants designed to address the "dying ReLU" problem. Here is a brief overview.

In summary, ReLU and its family offered a powerful solution to the vanishing gradient problem, enabling the deep learning revolution. ReLU remains a very common and effective default choice today.

The Modern Standard for Transformers: GELU

While ReLU is powerful, modern architectures like the Transformer (which powers models like GPT and BERT) have converged on a smoother, more complex activation function: the Gaussian Error Linear Unit (GELU). Its design is motivated by combining the properties of ReLU with the stochastic regularization of dropout.

The exact formula for GELU involves the Gaussian error function (erf), which is computationally expensive. In practice, a very accurate and fast approximation is used. For a more rigorous look at its derivation and properties, we will turn to the original paper that proposed it.

The key insight is that GELU acts as a smooth, probabilistic version of ReLU.

As the plots show, for positive inputs, GELU closely tracks the identity function, similar to ReLU. For negative inputs, it gently curves towards zero but doesn't have the hard zero "kink" of ReLU. Its derivative is smooth and continuous, peaking above 1 and having a small negative lobe, which contrasts sharply with ReLU's simple step-function derivative. These properties contribute to more stable and effective training in large-scale models.

Conclusion

We have journeyed from the historical sigmoid function to the modern GELU, seeing how the evolution of activation functions has been driven by the need to solve critical problems in training dynamics, particularly the vanishing gradient problem.

Key Takeaways:

• Sigmoid and Tanh are "saturating" functions whose gradients vanish for large inputs, making it difficult to train deep networks. Tanh is an improvement over Sigmoid because its output is zero-centered.
• ReLU provided a major breakthrough with its non-saturating positive part and computational efficiency. However, it can suffer from the "dying ReLU" problem where neurons become permanently inactive.
• GELU is a smooth, differentiable approximation to ReLU, motivated by stochastic regularization. Its non-monotonic shape and smooth gradient are thought to contribute to its superior performance in state-of-the-art Transformer models.

Here is a summary table for comparison:

In our next lesson, we will step back from the practical properties of these functions to address a more fundamental question: why does introducing any of these nonlinearities give neural networks their power? We will explore the Universal Approximation Theorem, which provides the theoretical foundation for the expressive capacity of neural networks.