Logistic Regression for Binary Classification

1. Why Linear Regression Isn't Right for Classification

Let's start by building a strong intuition for why we need a new algorithm. If we have data for a yes/no outcome (coded as 1 and 0), we could try to fit a linear regression line through it. But this approach has two major flaws.

As you saw, a simple straight line is not a good model for a yes/no outcome. We need a function that is "S-shaped" and stays neatly between 0 and 1.

2. The Sigmoid Function: From Linear Equation to Probability

This is where logistic regression gets its name. It combines the linear equation we already know with a special function called the logistic function, or more commonly, the sigmoid function.

The process is a two-step waltz:

1. Calculate a linear score: Just like in linear regression, we calculate a score based on a weighted sum of the features. We'll call this score $z$.
   $z={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_p{x}_p$

2. Apply the sigmoid function: We pass this score $z$ (which can be any real number) into the sigmoid function to transform it into a probability $p$ between 0 and 1.
   $p=\text{sigmoid}(z)=\frac{1}{1+e^{-z}}$

The output, $p$, is the predicted probability that the observation belongs to the positive class (class "1"). To make a final decision, we use a threshold (typically 0.5):

• If $p\ge 0.5$, we predict the class is 1.
• If $p<0.5$, we predict the class is 0.

To dive a bit deeper into the mathematics and see the sigmoid function's formula, the following resources are excellent.

3. Under the Hood: How Logistic Regression Learns

So, how does the model find the best coefficients (${\beta }_0,{\beta }_1,\dots$)? In linear regression, the goal was to minimize the sum of squared errors. For logistic regression, we use a different loss function called Binary Cross-Entropy, or Log Loss.

The intuition is simple:

• If the true label is 1, the loss is low if the predicted probability is close to 1, and high if it's close to 0.
• If the true label is 0, the loss is low if the predicted probability is close to 0, and high if it's close to 1.

The model then uses an optimization algorithm called Gradient Descent to iteratively adjust the coefficients to minimize this total loss across all training examples.

Given your interest in coding and how things work from scratch, the following video gives a fantastic mathematical walkthrough of deriving the loss function and implementing gradient descent. While you won't need to code this from scratch every time, understanding the mechanics is invaluable.

4. Implementing Logistic Regression with Scikit-Learn

Now, let's put theory into practice. Your experience with Python and scikit-learn from the previous lesson will be directly applicable here. The API is consistent and intuitive.

We will follow a similar workflow:

1. Import the necessary classes.
2. Split the data.
3. Create an instance of the LogisticRegression model.
4. Fit the model on the training data.
5. Make predictions on the test data.

The following video walks through a full example using the same insurance dataset from the first video.

Let's look at a code block summarizing this process. Note the introduction of model.predict_proba().

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression




# 1. Load and Prepare Data
# Dataset of customer age and whether they bought insurance (1=yes, 0=no)
data = {'age': [22, 25, 47, 52, 46, 56, 55, 60, 62, 61],
        'bought_insurance': [0, 0, 1, 0, 1, 1, 0, 1, 1, 1]}
df = pd.DataFrame(data)

X = df[['age']]
y = df['bought_insurance']




# 2. Split Data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)




# 3. Create and Fit the Model
model = LogisticRegression()
model.fit(X_train, y_train)




# 4. Make Predictions
# Let's test with some new ages
new_ages = [[28], [58]] # Must be a 2D array
predictions = model.predict(new_ages)
probabilities = model.predict_proba(new_ages)

print(f"Prediction for age 28: Class {predictions[0]}")
print(f"Probabilities for age 28: [P(class 0), P(class 1)] = {probabilities[0]}")
print("-" * 30)
print(f"Prediction for age 58: Class {predictions[1]}")
print(f"Probabilities for age 58: [P(class 0), P(class 1)] = {probabilities[1]}")




# 5. Evaluate the model on the test set
accuracy = model.score(X_test, y_test)
print(f"\nModel accuracy on the test set: {accuracy:.2f}")

The predict() method gives you the final class (0 or 1) based on the 0.5 threshold, while predict_proba() gives you the raw probabilities, which can be very useful for understanding the model's confidence.

For a comprehensive text resource covering implementation, best practices, and common pitfalls, the article we looked at earlier is an excellent reference.

Conclusion

Excellent work! You've just added a powerful and widely-used classification algorithm to your toolkit. You've moved from predicting continuous numbers to predicting categorical outcomes.

Key Takeaways:

• Logistic regression is used for binary classification problems.
• It works by passing a linear combination of features through a sigmoid function, which outputs a probability between 0 and 1.
• A decision threshold (usually 0.5) is used to convert the probability into a final class prediction (0 or 1).
• The model learns by minimizing a loss function called Binary Cross-Entropy using an algorithm like gradient descent.
• In scikit-learn, model.predict() gives the class label, and model.predict_proba() gives the class probabilities.

So far, we have looked at two linear models: linear regression and logistic regression. In our next lesson, we will explore a completely different approach to making predictions by using decision trees. This will introduce a new way of thinking about feature importance and model interpretability.