Understanding Logistic Regression and the Sigmoid Function

Why Linear Regression Fails for Classification

Let's start with a thought experiment. Imagine we have data on students, mapping hours studied to whether they passed (1) or failed (0) an exam. Could we just fit a standard linear regression line to this data?

At first, it might seem plausible. A line could trend upwards, suggesting that more study hours lead to a higher "score." However, this approach has two critical flaws.

To see these flaws clearly, let's watch a short video that sets up this exact problem.

As the video explained, the two major issues are:

1. Unbounded Outputs: A linear regression model can output any real number. It might predict a value of 1.3 or -0.2. What does it mean to have a 130% chance of passing, or a -20% chance? These values are not interpretable as probabilities, which should always be between 0 and 1.
2. Poor Fit: The straight line of a linear model doesn't accurately represent the nature of classification data, which often clusters at the extremes (0 and 1). A simple outlier can drastically change the slope of the line, shifting the decision point in a non-intuitive way.

We need a model that is specifically designed to output a probability. This is where logistic regression comes in.

The Heart of Logistic Regression: The Sigmoid Function

Logistic regression solves the problems of linear regression by introducing a new component: the sigmoid function, also known as the logistic function.

The core idea is simple yet powerful:

1. We start with the same linear equation as before: $z=\mathbf{w}\cdot \mathbf{x}+b$. This combines our features ($\mathbf{x}$) and weights ($\mathbf{w}$) to produce a score, $z$. This score can be any real number.
2. Instead of using $z$ as the final output, we pass it through the sigmoid function, which "squashes" the value into a range between 0 and 1.

The sigmoid function is defined as:
$\sigma (z)=\frac{1}{1+e^{-z}}$

This "S"-shaped curve is the key to transforming the unbounded output of a linear equation into a meaningful probability.

Let's watch a brief, intuitive explanation of how this works in practice.

The output of the sigmoid function, $\sigma (z)$, is interpreted as the probability that the given input belongs to the positive class (often labeled '1'). For our student example, it's the probability of passing the exam.

• If $z$ is a large positive number (e.g., many hours studied), $e^{-z}$ approaches 0, and $\sigma (z)$ approaches $\frac{1}{1+0}=1$.
• If $z$ is a large negative number, $e^{-z}$ becomes very large, and $\sigma (z)$ approaches 0.
• If $z=0$, then $e^{-0}=1$, and $\sigma (z)=\frac{1}{1+1}=0.5$.

So, the logistic regression model's prediction is formally written as:
$\hat{p}=P(y=1|\mathbf{x})=\sigma (\mathbf{w}\cdot \mathbf{x}+b)=\frac{1}{1+e^{-(\mathbf{w}\cdot \mathbf{x}+b)}}$
where $\hat{p}$ is the predicted probability that the outcome $y$ is 1, given the input features $\mathbf{x}$.

From Probability to a Final Decision

The model now gives us a probability, but our goal is to make a definite classification (e.g., "Pass" or "Fail"). To do this, we set a decision boundary, which is a threshold for the probability.

The most common threshold is 0.5. The rule is simple:

• If $\hat{p}\ge 0.5$, we predict Class 1 (Pass).
• If $\hat{p}<0.5$, we predict Class 0 (Fail).

Because the sigmoid function outputs 0.5 when its input is 0, this threshold directly relates back to our linear equation:

• Predict Class 1 when $\mathbf{w}\cdot \mathbf{x}+b\ge 0$.
• Predict Class 0 when $\mathbf{w}\cdot \mathbf{x}+b<0$.

The line defined by $\mathbf{w}\cdot \mathbf{x}+b=0$ is the decision boundary that separates the two classes in the feature space.

The Mathematical Origin of the Sigmoid Function

You have a preference for understanding the theory first, so you might be asking: why this specific S-shaped function? Where does it come from? The choice is not arbitrary; it's derived from a linear model of log-odds.

Let's break this down.

1. Probability (p): The probability of an event happening. Range: [0, 1].
2. Odds: The ratio of the probability of the event happening to it not happening. Odds = $\frac{p}{1-p}$. Range: [0, $\infty$).
3. Log-odds (or logit): The natural logarithm of the odds. Log-odds = $\log \left(\frac{p}{1-p}\right)$. Range: ($-\infty ,\infty$).

Notice that the log-odds are unbounded, just like the output of a linear equation. The core assumption of logistic regression is that the log-odds of the outcome are a linear function of the input features.

$\log \left(\frac{p}{1-p}\right)=\mathbf{w}\cdot \mathbf{x}+b$

This elegant assumption connects the unbounded linear model to our problem. If we now solve this equation algebraically for $p$, we get back precisely the sigmoid function!

For a more detailed look at the mathematics, please read the following sections.

This derivation is a beautiful example of how a constrained problem (predicting probabilities between 0 and 1) can be solved by transforming it into an unconstrained space (log-odds), applying a simple linear model, and then transforming the result back.

Conclusion

In this lesson, you have transitioned from regression to classification and learned the theoretical underpinnings of logistic regression. This algorithm is a cornerstone of machine learning, serving not only as a powerful classification tool in its own right but also as a building block for more complex models like neural networks.

Key Takeaways:

• Classification is the task of predicting a discrete category, unlike regression, which predicts a continuous value.
• Linear regression is unsuitable for classification because its output is unbounded and not easily interpreted as a probability.
• Logistic regression solves this by using a linear model ($z=\mathbf{w}\cdot \mathbf{x}+b$) as the input to the sigmoid function.
• The sigmoid function, $\sigma (z)=\frac{1}{1+e^{-z}}$, "squashes" any real-valued input $z$ into a probability between 0 and 1.
• The model's output is the probability of the input belonging to the positive class (1). A decision threshold (typically 0.5) is then used to make the final classification.
• The sigmoid function is mathematically derived from the assumption that the log-odds of the outcome are a linear function of the features.

Preview of the Next Lesson:
Now that you have a solid grasp of the theory, it's time to put it into practice. In the next lesson, you will train and evaluate your first logistic regression model using the scikit-learn library in Python, applying these concepts to a real dataset.