Logistic Regression with Scikit-learn

Hello! Welcome to the fourth lesson in our course.

In our last lesson, you successfully built and evaluated a linear regression model. You practiced the complete machine learning workflow: preparing data, splitting it into training and test sets, fitting a model, and using metrics like R-squared and MSE to measure its performance on a regression task.

Today, we shift our focus from predicting continuous numbers to predicting categories. This is known as classification, a cornerstone of machine learning. We will build upon the Scikit-learn workflow you've learned to train and evaluate a logistic regression classifier.

By the end of this lesson, you will be able to:

• Explain the role of the sigmoid function in logistic regression.
• Train a logistic regression model using Scikit-learn's standard fit/predict pattern.
• Evaluate the model's performance using classification-specific metrics like accuracy, the confusion matrix, precision, and recall.

This lesson will solidify your practical skills and introduce you to the essential concepts and tools for evaluating any classification model you build in the future.

1. From Linear to Logistic: The Core Idea

In the last lesson, we used linear regression to fit a straight line to our data. What if our goal isn't to predict a price, but to classify an outcome, like whether a customer will buy a product (Yes/No) or if an email is spam (Spam/Not Spam)? These are binary classification problems.

A natural first thought might be to use linear regression anyway and set a threshold. For example, if the output is > 0.5, predict 'Yes'; otherwise, predict 'No'. However, this approach has significant flaws.

This short video provides an excellent visual explanation of why linear regression is not suitable for classification and introduces the key component of logistic regression that solves this problem.

As the video explained, the magic of logistic regression lies in the sigmoid function. It takes any real-valued number and maps it to a value between 0 and 1.

The equation for logistic regression combines the linear equation you're familiar with and the sigmoid function:

1. First, calculate the linear part (called the logit):
   $z=w_0+w_1{x}_1+\cdots +w_k{x}_k$
2. Then, pass this result through the sigmoid function to get a probability, $p$:
   $p=\frac{1}{1+e^{-z}}$

This probability $p$ represents the model's confidence that the given input belongs to the positive class (labeled '1').

2. Building a Logistic Regression Model in Scikit-learn

Now let's apply this theory. The great news is that the Scikit-learn workflow for training a logistic regression model is nearly identical to the one you used for linear regression. The main steps are initialize -> fit -> predict.

The following video provides a clear, practical demonstration of this process.

As you can see, the core implementation pattern remains consistent, which is a key design principle of Scikit-learn.

3. Evaluating a Classification Model

For linear regression, we used metrics like MSE and R-squared. For classification, we need different metrics to assess performance. While accuracy is a starting point, it doesn't tell the whole story, especially when one class is much more common than the other (an "imbalanced dataset").

A much more informative tool is the confusion matrix.

The Confusion Matrix

The confusion matrix gives a 2x2 table that breaks down the model's predictions into four categories:

• True Positives (TP): The model correctly predicted the positive class (e.g., predicted 'Yes', and it was 'Yes').
• True Negatives (TN): The model correctly predicted the negative class (e.g., predicted 'No', and it was 'No').
• False Positives (FP): The model incorrectly predicted the positive class (a "Type I error").
• False Negatives (FN): The model incorrectly predicted the negative class (a "Type II error").

From these four values, we can derive more nuanced metrics:

• Accuracy: How many predictions were correct overall?
  $\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$
• Precision: Of all the times the model predicted positive, how many were correct? (Measures the quality of positive predictions).
  $\text{Precision}=\frac{TP}{TP+FP}$
• Recall (Sensitivity): Of all the actual positive cases, how many did the model find? (Measures the model's ability to find all positive instances).
  $\text{Recall}=\frac{TP}{TP+FN}$
• F1-Score: The harmonic mean of Precision and Recall. It provides a single score that balances both metrics.
  $\text{F1-Score}=2\times \frac{\text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$

The following reading provides a detailed walkthrough of how to generate and interpret these metrics using Scikit-learn.

This article provides all the necessary code and interpretation to move beyond simple accuracy and robustly evaluate your classification models.

Key Takeaways

• Logistic Regression is a linear model used for binary classification. It uses the sigmoid function to output a probability between 0 and 1.
• The Scikit-learn workflow (train_test_split, model.fit(), model.predict()) is consistent across different model types, making it easy to switch between them.
• Accuracy is a simple evaluation metric, but it can be misleading for imbalanced datasets.
• The Confusion Matrix provides a detailed breakdown of correct and incorrect predictions (TP, TN, FP, FN).
• Precision, Recall, and F1-Score are crucial metrics derived from the confusion matrix that give a more complete picture of a classifier's performance. Scikit-learn's classification_report is the go-to function for this.

What's Next?

Fantastic job! You have now trained and evaluated both a regression and a classification model, covering two fundamental areas of supervised learning.

While powerful, logistic regression can only create a linear decision boundary between classes. In the next lesson, we will explore tree-based models like Decision Trees and Random Forests, which can capture more complex, non-linear patterns in data. You will see how these models work and, once again, apply the same evaluation techniques you learned today.