Implementing Common Activation Functions

Why Do We Need Activation Functions?

In the previous lesson, you used the sigmoid function to convert log-odds into probabilities. In a neural network, activation functions serve a broader and more fundamental purpose: they introduce non-linearity.

A single neuron in a network first computes a weighted sum of its inputs plus a bias—a linear operation. If we were to stack layers of these neurons without an activation function, the entire network would still just be a single, complex linear function. It would be mathematically equivalent to a simple linear model, incapable of learning complex patterns like the ones in your computer vision projects.

Activation functions are applied to the output of the weighted sum, transforming it non-linearly. This allows networks with multiple layers to approximate incredibly complex, non-linear relationships between inputs and outputs.

To get a clear visual and conceptual overview of this, please watch the first few minutes of the following video.

Now that you understand why we need them, let's implement the most common ones. We'll use a geeksforgeeks article as our primary implementation guide, as it provides clear code and explanations.

1. The Sigmoid Function

You are already familiar with the Sigmoid function from logistic regression and gradient boosting classification. It takes any real-valued number and "squashes" it into a range between 0 and 1. This makes it particularly useful for the output layer of a binary classification model, where the output can be interpreted as a probability.

Formula:
$\sigma (x)=\frac{1}{1+e^{-x}}$

Implementation

Let's implement and visualize the Sigmoid function. The geeksforgeeks article provides the code to do this.

Follow Step 1 through Step 5 in the "Step-By-Step Implementation" section of the article you just opened. This will guide you to:

1. Import numpy and matplotlib.
2. Define the sigmoid function.
3. Generate an input range of data x.
4. Compute the output y_sigmoid.
5. Plot the result.

Your Python code for the function should look like this:

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Generate input data
x = np.linspace(-10, 10, 100)

# Compute sigmoid output
y_sigmoid = sigmoid(x)

# Plotting
plt.figure(figsize=(8, 6))
plt.plot(x, y_sigmoid, label='Sigmoid')
plt.title("Sigmoid Activation Function")
plt.xlabel("Input (x)")
plt.ylabel("Output σ(x)")
plt.grid(True)
plt.show()

You should see a characteristic 'S'-shaped curve that saturates at 0 and 1.

While useful, Sigmoid functions have a major drawback in hidden layers: the vanishing gradient problem. For very large positive or negative inputs, the curve becomes flat. A flat curve means the gradient (derivative) is close to zero. During training, these near-zero gradients can cause the learning process to become extremely slow or stop altogether.

2. The Tanh (Hyperbolic Tangent) Function

The Tanh function is another S-shaped curve, but it squashes values to a range between -1 and 1.

Formula:
$\tanh (x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$

The main advantage of Tanh over Sigmoid is that its output is zero-centered. This property often helps the model learn more efficiently during training. However, like Sigmoid, it also suffers from the vanishing gradient problem when inputs become large.

Implementation

Now, let's implement Tanh. Follow Step 6 in the geeksforgeeks article to plot the Tanh function.

The Python implementation can use NumPy's built-in np.tanh for simplicity and efficiency.

def tanh(x):
    return np.tanh(x)

# Compute tanh output
y_tanh = tanh(x) # Using the same 'x' from before

# Plotting
plt.figure(figsize=(8, 6))
plt.plot(x, y_tanh, label='Tanh', color='orange')
plt.title("Tanh Activation Function")
plt.xlabel("Input (x)")
plt.ylabel("Output tanh(x)")
plt.grid(True)
plt.show()

The plot will show an S-shaped curve that ranges from -1 to 1 and passes through the origin (0,0).

3. The ReLU (Rectified Linear Unit) Function

ReLU is arguably the most popular activation function used in the hidden layers of neural networks today. It is simple, computationally efficient, and helps mitigate the vanishing gradient problem.

The function is very straightforward:

• If the input is positive, the output is the input itself.
• If the input is negative, the output is zero.

Formula:
$f(x)=\max (0,x)$

This simple operation avoids the expensive exponential calculations of Sigmoid and Tanh. More importantly, for all positive inputs, its gradient is a constant 1, allowing gradients to flow much more effectively during training and leading to faster learning.

Implementation

The implementation of ReLU is elegantly handled by NumPy's maximum function. Follow Step 7 in the geeksforgeeks article to implement and plot ReLU.

def relu(x):
    return np.maximum(0, x)

# Compute ReLU output
y_relu = relu(x) # Using the same 'x' from before

# Plotting
plt.figure(figsize=(8, 6))
plt.plot(x, y_relu, label='ReLU', color='green')
plt.title("ReLU Activation Function")
plt.xlabel("Input (x)")
plt.ylabel("Output ReLU(x)")
plt.grid(True)
plt.show()

The resulting plot shows a function that is zero for all negative inputs and then increases linearly with a slope of 1 for all positive inputs.

A drawback of ReLU is the "dying ReLU" problem: if a neuron's input is consistently negative, it will always output zero. Consequently, its gradient will also always be zero, and it will stop learning entirely. A common variant called Leaky ReLU addresses this by allowing a small, non-zero gradient for negative inputs (e.g., $f(x)=0.01x$ for $x<0$), but standard ReLU remains the most common starting point.

Summary and Comparison

You have now implemented the three most fundamental activation functions. Each has its own characteristics and typical use cases.

The geeksforgeeks article provides a great summary table. Review the section "Difference Between Sigmoid, Tanh and ReLU" to see a side-by-side comparison of their range, gradient issues, computational cost, and ideal use cases.

Here is a visual summary of the functions you've built:

Conclusion

In this lesson, you've taken the first crucial step into building neural networks. You learned that activation functions are essential for introducing the non-linearity that allows networks to learn complex patterns. You then implemented the three most common activation functions—Sigmoid, Tanh, and ReLU—from scratch, understanding their mathematical definitions, properties, and trade-offs.

Key Takeaways:

• Non-linearity: Activation functions are what enable deep neural networks to model non-linear phenomena. Without them, a multi-layer network collapses into a simple linear model.
• Sigmoid: Squashes output to a [0, 1] range. Ideal for binary classification output layers but suffers from vanishing gradients in hidden layers.
• Tanh: Squashes output to a [-1, 1] range. Its zero-centered nature is an advantage over Sigmoid, but it also has a vanishing gradient problem.
• ReLU: Outputs max(0, x). It is computationally cheap and avoids vanishing gradients for positive inputs, making it the default choice for hidden layers in modern networks.

Preview of the Next Lesson:
Now that we have the activation functions—the core processing unit of a neuron—we need to initialize the parameters that feed into them: the weights and biases. In the next lesson, you will implement weight initialization strategies, including random and Xavier/He initialization, to ensure our network starts its learning process on the right foot.