Implementing and Visualizing the Sigmoid Function

1. From Linear Output to Probability

Think back to your linear regression model. Its output was a linear combination of features, ${\theta }^{T}x$, which could result in any number from negative to positive infinity. This is perfect for predicting prices or temperatures, but what if we want to answer a binary question?

• Is this email spam or not? (0 or 1)
• Will this customer churn or not? (0 or 1)

We need a way to take the continuous output of a linear model and map it to a probability, a value between 0 and 1. This is precisely the job of the sigmoid function.

The sigmoid function, often denoted by $\sigma (z)$, is defined by the following formula:

$\sigma (z)=\frac{1}{1+e^{-z}}$

Where $z$ would be the output of our linear model ($z={\theta }^{T}x$).

The function's 'S'-shaped curve (which you'll soon plot yourself) takes any real number $z$ and squashes it into the (0, 1) range.

• If $z$ is a large positive number, $e^{-z}$ becomes very small, so $\sigma (z)\approx 1$.
• If $z$ is a large negative number, $e^{-z}$ becomes very large, so $\sigma (z)\approx 0$.
• If $z=0$, $e^0=1$, so $\sigma (0)=1/(1+1)=0.5$.

This behavior allows us to interpret the output of the sigmoid function as the probability of the positive class. For example, if our model outputs 0.8, we can say there's an 80% probability that the sample belongs to class 1.

2. Implementing the Sigmoid Function From Scratch

Now, let's turn theory into practice. Your goal is to write a Python function that implements the sigmoid formula. Given your background, you'll appreciate how NumPy's vectorization capabilities make this incredibly efficient.

The following resource provides a complete, self-contained guide to implementing and visualizing the sigmoid function. It aligns perfectly with your preference for "from scratch" coding.

Let's break down the implementation and visualization code from the resource.

Step 1: The Implementation

Create a new Python script or Jupyter Notebook. First, import the necessary libraries:

import numpy as np
import matplotlib.pyplot as plt

Now, define the sigmoid function. Using np.exp() allows this function to work on both single numbers and entire NumPy arrays (a concept known as vectorization). This is far more efficient than looping through an array in Python.

def sigmoid(z):
  """
  Computes the sigmoid of z.
  
  Args:
    z (float or np.ndarray): The input value(s).
    
  Returns:
    (float or np.ndarray): The sigmoid of z.
  """
  return 1 / (1 + np.exp(-z))

You can test it with a few values:

print(f"Sigmoid of 100: {sigmoid(100)}")   # Should be very close to 1.0
print(f"Sigmoid of 0: {sigmoid(0)}")     # Should be exactly 0.5
print(f"Sigmoid of -100: {sigmoid(-100)}") # Should be very close to 0.0

# Test with a NumPy array
test_array = np.array([-10, -2, 0, 2, 10])
print(f"Sigmoid on an array: {sigmoid(test_array)}")

Step 2: The Visualization

Seeing the function's curve makes its behavior intuitive. We'll generate a range of input values and plot their corresponding sigmoid outputs.

# 1. Generate a range of input values
z_values = np.linspace(-10, 10, 200) # 200 points from -10 to 10

# 2. Compute the sigmoid for these values
sigmoid_values = sigmoid(z_values)

# 3. Plot the results
plt.figure(figsize=(10, 6))
plt.plot(z_values, sigmoid_values, label='Sigmoid Function')
plt.title('The Sigmoid Function')
plt.xlabel('z (Input)')
plt.ylabel('sigmoid(z) (Output Probability)')
plt.grid(True)
plt.axhline(y=0.5, color='r', linestyle='--', label='y = 0.5')
plt.axvline(x=0, color='r', linestyle='--')
plt.legend()
plt.show()

When you run this code, you will see the classic 'S'-shaped curve. Notice how it's flat for large positive and negative inputs (this is called saturation) and steepest around $z=0$. This steep region is where the model is most "uncertain" and small changes in the input $z$ lead to large changes in the output probability.

def custom_sigmoid(x):
    if x >= 0:
        z = math.exp(-x)
        return 1 / (1 + z)
    else:
        z = math.exp(x)
        return z / (1 + z)

Conclusion

Congratulations on implementing your first component of a classification algorithm! The sigmoid function is a simple but powerful tool that forms the bridge between linear models and probabilistic classification.

Key Takeaways:

• The sigmoid function maps any real-valued input to an output between 0 and 1.
• Its mathematical formula is $\sigma (z)=\frac{1}{1+e^{-z}}$.
• In classification, it's used to convert a model's raw output into a probability, which is essential for making decisions.
• Using NumPy allows for an efficient, vectorized implementation that can process entire arrays of data at once.

Preview of the Next Lesson:
You now have a function that can take a linear score, $z={\theta }^{T}x$, and produce a predicted probability, $\hat{y}=\sigma (z)$. The next logical question is: how do we find the best parameters ($\theta$)? To do that, we need a way to measure how "wrong" our predictions are. In the next lesson, you will derive and calculate the gradient of the binary cross-entropy loss function, which is the cost function used to train logistic regression.