Logistic Regression for Binary Classification

Hello! Welcome to our new module, "Classification Models and Evaluation."

In the previous module, we focused entirely on regression—predicting continuous numerical values like price or temperature. We culminated in learning how to use k-fold cross-validation to tune hyperparameters for models like Ridge and Lasso. Now, we're making a fundamental shift from "how much?" to "which one?". We're moving into the world of classification.

This lesson kicks off our journey into classification by tackling the simplest yet most common type: binary classification, where the outcome has only two possibilities (e.g., "spam" or "not spam," "pass" or "fail"). Our goal is to model binary classification probabilities using the logistic (sigmoid) function. You'll discover how we can cleverly adapt the linear models you already know to handle this new type of problem.

Why Can't We Just Use Linear Regression?

Let's imagine we want to predict whether a student will pass (1) or fail (0) an exam based on the hours they studied. This seems similar to the problems we solved with linear regression. So, why not just fit a straight line to the data?

Let's explore the limitations of this approach.

Logistic Regression (and why it's different from Linear Regression)

To understand why linear regression isn't suitable for classification, watch this segment from the video 'Logistic Regression' by Visually Explained. It clearly illustrates the problems that arise when you try to fit a straight line to binary outcomes.

Watch from the beginning to 02:41. Pay close attention to two key issues: 1) The linear regression line can predict values greater than 1 or less than 0, which are meaningless as probabilities. 2) The model's predictions are not constrained to the valid [0, 1] range required for probabilities.

As the video shows, using a linear model directly for a binary classification task has two major flaws:

Nonsensical Predictions: The model can output values like 1.3 or -0.2. What does a 130% or -20% probability of passing mean? It's undefined.
Poor Fit: The relationship between study hours and passing isn't truly linear. It's more of a threshold effect—at some point, the probability of passing rapidly increases. A straight line doesn't capture this S-shaped relationship well.

We need a function that takes the output of a linear model and "squashes" it into a valid probability, i.e., a value between 0 and 1.

The Solution: The Logistic (Sigmoid) Function

The function that elegantly solves this problem is the sigmoid function, also known as the logistic function. Its job is to take any real number and map it to the range (0, 1).

The core idea of logistic regression is a two-step process:

Calculate a score using a linear equation, just like in linear regression. Let's call this score $z$ . $z = b + w_{1} x_{1} + w_{2} x_{2} + \dots + w_{N} x_{N}$ Here, $x_{i}$ are your features, $w_{i}$ are the weights, and $b$ is the bias.
Feed this score $z$ into the sigmoid function to transform it into a probability. $P (y = 1) = σ (z) = \frac{1}{1 + e ^{- z}}$

This process is summarized visually in the following diagram.

Sigmoid (Logistic) Function for Binary Classification — This diagram illustrates the core mechanism of logistic regression. The input features (x) are first combined linearly with weights (w) and a bias (b) to produce a score 'z'. This score 'z' is then passed through the sigmoid function, which transforms it into a probability between 0 and 1.

To understand the sigmoid function better, let's explore its properties and see how it works.

Calculating a probability with the sigmoid function

The Google Machine Learning Crash Course offers a very clear and concise explanation of the sigmoid function and its role in logistic regression. This reading will solidify your understanding of the formula and its behavior.

Please read the sections titled 'Sigmoid function' and 'Transforming linear output using the sigmoid function'. Focus on the formula for the sigmoid, its characteristic S-shaped graph, and how it maps the linear output z to a probability y'.

The key takeaway is that no matter how large or small the value of $z$ is, the output of the sigmoid function will always be between 0 and 1, making it perfect for representing probability.

When we plot this function over our data, it produces the S-shaped curve we intuitively expected.

Sigmoid Function for Binary Classification Example — This graph shows how the sigmoid function models the probability of passing a test based on learning hours. The red and green dots represent actual outcomes (fail and pass). The blue S-shaped curve is the output of the logistic regression model, representing the predicted probability of passing for any given number of hours.

Putting It All Together: A Calculation Example

Let's walk through a concrete example. Suppose we have a trained logistic regression model that predicts whether an email is spam. The model has two features:

$x_{1}$ : the number of capitalized words.
$x_{2}$ : the presence of the word "free" (1 if present, 0 if not).

The model has learned the following parameters:

Bias: $b = - 3$
Weight for $x_{1}$ : $w_{1} = 0.1$
Weight for $x_{2}$ : $w_{2} = 2$

Now, a new email arrives with 10 capitalized words ( $x_{1} = 10$ ) and it contains the word "free" ( $x_{2} = 1$ ). What is the probability that this email is spam?

Step 1: Calculate the linear score, $z$ .

z = b + w_{1} x_{1} + w_{2} x_{2} = - 3 + (0.1 \times 10) + (2 \times 1)

z = - 3 + 1 + 2 = 0

Step 2: Apply the sigmoid function to $z$ .

P (spam) = σ (z) = \frac{1}{1 + e ^{- z}} = \frac{1}{1 + e ^{- 0}} = \frac{1}{1 + 1} = 0.5

So, the model predicts a 50% probability that this email is spam.

Now, it's your turn to practice this calculation.

Test your understanding!

A logistic regression model with three features has the following bias and weights:

$b = 1$
$w_{1} = 2$
$w_{2} = - 1$
$w_{3} = 5$

You are given an example with the following input values:

$x_{1} = 0$
$x_{2} = 10$
$x_{3} = 2$

First, calculate the value of $z$ . Then, calculate the final logistic regression prediction (the probability).

Show answer

1. Calculate z:

z = b + w_{1} x_{1} + w_{2} x_{2} + w_{3} x_{3} = 1 + (2 \times 0) + (- 1 \times 10) + (5 \times 2)

z = 1 + 0 - 10 + 10 = 1

The value of $z$ is 1.

2. Calculate the prediction:

Prediction = σ (1) = \frac{1}{1 + e ^{- 1}} \approx \frac{1}{1 + 0.3678} = \frac{1}{1.3678} \approx 0.731

The predicted probability is approximately 0.731 or 73.1%.

(This is the same exercise found in the Google reading, confirming your understanding.)

For an additional perspective and to see a simple implementation in Python, you can review the following resource.

Logistic Regression in Machine Learning

The article 'Logistic Regression in Machine Learning' from GeeksForGeeks provides a solid summary of the concepts we've covered, including the sigmoid function and the overall workflow. It also shows how to implement it with scikit-learn.

Read the sections 'Understanding Sigmoid Function', 'How does Logistic Regression work?', and 'Logistic Regression Equation and Odds'. This will reinforce the key ideas and show the mathematical derivation from log-odds to the sigmoid formula, which is a great preview for our next lesson.

Conclusion

In this lesson, you've made the crucial leap from regression to classification. You learned why linear regression is unsuitable for predicting binary outcomes and how logistic regression solves this by using the sigmoid function to map a linear model's output to a valid probability.

Key Takeaways:

Linear regression is not appropriate for classification because its output is unbounded and doesn't represent probability.
Logistic regression models the probability of a binary outcome.
It achieves this by passing the output of a linear model ( $z = w \cdot X + b$ ) through the sigmoid function ( $σ (z)$ ).
The sigmoid function, $σ (z) = \frac{1}{1 + e ^{- z}}$ , squashes any real number into the range (0, 1).

Preview of the Next Lesson:
We now know how to calculate a probability with logistic regression. But what do the model's weights ( $w_{1}, w_{2}, \dots$ ) actually mean? In the next lesson, we will learn how to interpret logistic regression coefficients in terms of log-odds ratios. This will allow us to understand how each feature influences the probability of the outcome, moving us closer to our goal of justifying our modeling choices.

Recommended resources

Why Can't We Just Use Linear Regression?

The Solution: The Logistic (Sigmoid) Function

Putting It All Together: A Calculation Example

Conclusion