Demystifying Logistic Regression

From Linear to Logistic: The Need for a New Approach

Let's start with a simple question: why can't we just use the linear regression models we've already mastered for a classification task?

Imagine a binary classification problem where we want to predict whether a tumor is malignant (1) or benign (0) based on its size. If we try to fit a standard linear regression line to this data, we immediately run into a problem.

As you can see, the linear model's predictions can extend far beyond the logical bounds of 0 and 1. What does a prediction of 1.5 or -0.2 mean when our labels are only 0 or 1? This output is not easily interpretable as a probability. We need a way to take the output of a linear equation and "squash" it into the [0, 1] range. This is where the sigmoid function comes in.

The Sigmoid Function: Mapping Output to Probability

The core innovation of logistic regression is the use of the sigmoid function (also called the logistic function) to transform the output. The sigmoid function takes any real-valued number and maps it to a value between 0 and 1, making it perfect for modeling probability.

Its mathematical form is:

Here, $z$ is simply the output of the familiar linear equation: $z=\mathbf{w}\cdot \mathbf{x}+b$, where $\mathbf{w}$ are the model weights and $b$ is the bias term.

The video below from the StatQuest channel provides a wonderful high-level explanation of how this works in practice.

The sigmoid function elegantly solves our problem. No matter how large or small the output of our linear equation $z$ is, the resulting probability $\sigma (z)$ will always be between 0 and 1.

Log-Odds: The "Linear" Core of Logistic Regression

We've established that logistic regression uses a linear model inside a sigmoid function to output a probability. But what is that inner linear model actually representing? It's not modeling the probability directly. Instead, it models the log-odds of the event occurring.

This is often the most confusing part of logistic regression, so let's break it down carefully. The StatQuest channel has another excellent video that clarifies the concepts of odds and log-odds.

As you just saw, the logit function is $\log \left(\frac{p}{1-p}\right)$. This is what the linear model at the heart of logistic regression predicts.

So, the full relationship is:

This equation reveals why logistic regression is a member of the family of "linear classifiers." While the relationship between the features $\mathbf{x}$ and the probability $p(\mathbf{x})$ is non-linear (due to the sigmoid curve), the relationship between the features $\mathbf{x}$ and the log-odds of the outcome is linear. By solving for $p(\mathbf{x})$ in the equation above, we arrive back at our sigmoid function.

For a more detailed look at the mathematics connecting these concepts, the following resource is very helpful.

The Decision Boundary: From Probability to Classification

We now have a model that outputs a probability. To make a final classification (e.g., "0" or "1"), we must define a decision boundary. This is typically done by setting a probability threshold, most commonly 0.5.

• If $p(\mathbf{x})\ge 0.5$, we predict class 1.
• If $p(\mathbf{x})<0.5$, we predict class 0.

Let's think about what a probability of 0.5 means in the context of our model.

• If $p=0.5$, then the odds are $\frac{0.5}{1-0.5}=1$.
• The log-odds are $\log (1)=0$.

This means the decision boundary corresponds to the exact point where our linear model's output is zero:

This equation defines a line in 2D, a plane in 3D, or a hyperplane in higher dimensions. This hyperplane is the decision boundary that linearly separates the two classes in the feature space. Any point on one side of the hyperplane is classified as 1, and any point on the other side is classified as 0.

This image beautifully illustrates the entire process:

The Real Python article provides excellent visualizations for both one-dimensional and two-dimensional cases, clearly showing this linear separation.

One final note: how does the model find the best weights ($\mathbf{w}$) and bias ($b$)? In linear regression, we used Ordinary Least Squares to minimize the sum of squared residuals. In logistic regression, this approach doesn't work. Instead, the model is fit using a method called Maximum Likelihood Estimation (MLE). You don't need to know the derivation, but the intuition is that MLE finds the parameters that maximize the probability of observing the actual data you have.

Conclusion

In this lesson, you have built the foundational intuition for logistic regression. We have moved from the world of continuous prediction to binary classification by understanding how to adapt a linear model for this new task.

Key Takeaways:

• Logistic regression is a classification algorithm that models the probability of a binary outcome.
• It uses the sigmoid function to "squash" the output of a linear equation into the range [0, 1].
• The underlying linear model does not predict the probability directly but rather the log-odds (logit) of the event. The relationship between features and log-odds is linear.
• A decision boundary, typically at a probability of 0.5 (which corresponds to log-odds of 0), is used to make the final classification. This boundary is a linear hyperplane in the feature space.

You now understand the "how" and "why" of logistic regression's mechanics. In our next lesson, we will discuss the practical side: when is it an appropriate algorithm to choose? We will then begin our deep dive into the essential metrics used to evaluate classification models, starting with the confusion matrix.