Logistic Regression with scikit-learn

The Standard Scikit-Learn Workflow

Scikit-learn is a cornerstone library for machine learning in Python. It provides a consistent and user-friendly interface for most common algorithms. The process for training a supervised learning model like logistic regression typically follows these five steps:

1. Prepare Data: Load your dataset and separate it into features (X) and the target variable (y).
2. Split Data: Divide your data into a training set (to teach the model) and a testing set (to evaluate its performance on unseen data).
3. Instantiate Model: Create an instance of the model class from scikit-learn (e.g., LogisticRegression()).
4. Train Model: "Fit" the model to the training data using the .fit() method. This is where the learning happens—the model finds the optimal weights (w) and bias (b).
5. Make Predictions: Use the trained model to make predictions on the test data with the .predict() method.

Let's see this entire workflow in action.

This sequence—import, instantiate, fit, predict—is a fundamental pattern you will use repeatedly as you build more complex machine learning models.

Evaluating Model Performance

Now that we have a trained model and a set of predictions, how do we know if it's any good? This "evaluation" step is arguably the most critical part of the process. A model is only useful if we can quantify and trust its performance.

The Baseline: Accuracy

The most intuitive metric is accuracy: the proportion of correct predictions out of the total number of predictions. Scikit-learn provides a convenient .score() method to calculate this.

While simple, accuracy can be dangerously misleading, especially for datasets where one class is much more frequent than another (an "imbalanced dataset"). For example, if you build a model to detect a rare disease (affecting 0.1% of the population) and it achieves 99.9% accuracy by always predicting "no disease," it's a perfect score but a completely useless model.

To make the kind of high-stakes decisions you're aiming for, we need more nuanced metrics.

A Deeper Look: The Confusion Matrix

A confusion matrix provides a much more detailed breakdown of a model's performance. It's a table that compares the actual values to the predicted values and separates them into four categories.

• True Positives (TP): The model correctly predicted the positive class. (e.g., correctly identified a patient with the disease).
• True Negatives (TN): The model correctly predicted the negative class. (e.g., correctly identified a patient without the disease).
• False Positives (FP): The model incorrectly predicted the positive class. (e.g., told a healthy patient they have the disease). This is also known as a "Type I error."
• False Negatives (FN): The model incorrectly predicted the negative class. (e.g., told a sick patient they are healthy). This is also known as a "Type II error."

From this matrix, we can derive more powerful metrics.

Precision, Recall, and F1-Score

• Precision: Answers the question, "Of all the times the model predicted positive, how often was it correct?"
  $\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}$
  High precision is critical when the cost of a False Positive is high. For example, in spam filtering, you want to be very precise to avoid putting an important email in the spam folder (a false positive).

• Recall (or Sensitivity): Answers the question, "Of all the actual positive cases, how many did the model correctly identify?"
  $\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}$
  High recall is critical when the cost of a False Negative is high. In medical diagnosis, you want high recall to avoid missing any actual cases of disease (a false negative).

• F1-Score: The harmonic mean of Precision and Recall. It provides a single score that balances both concerns.
  $\text{F1-Score}=2\times \frac{\text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$

The scikit-learn library makes it easy to compute all these metrics at once using a classification_report.

Understanding the trade-off between precision and recall is central to applied machine learning. Deciding whether it's worse to have a False Positive or a False Negative depends entirely on the problem context—a skill essential for the kind of advanced, pressure-tested decision-making you aim to master.

Fine-Tuning the Model

So far, we have used LogisticRegression() with its default settings. However, "training" a model also involves choosing the right configuration. These settings, known as hyperparameters, control the learning process itself.

Two of the most important hyperparameters for logistic regression are penalty and C. These control regularization, a technique to prevent overfitting by penalizing model complexity. You encountered this idea in Module 7 with the bias-variance tradeoff.

Here's the key relationship to remember for the hyperparameter C:

• Small C: Strong regularization. The model is pushed to be simpler, which can help prevent overfitting but might lead to underfitting.
• Large C: Weak regularization. The model is allowed to be more complex to fit the training data better, but this increases the risk of overfitting.

Finding the optimal values for these hyperparameters is a process called hyperparameter tuning, which is a more advanced topic we will revisit later. For now, it's important to know that these levers exist for you to control and optimize your model.

Conclusion

You have now successfully bridged the gap from theory to practice. You've learned the standard procedure for implementing a machine learning model in scikit-learn and, more importantly, how to critically evaluate its performance beyond simple accuracy.

Key Takeaways:

• The standard scikit-learn workflow is to prepare data, split, instantiate, fit, and predict.
• Accuracy is an easy-to-understand metric but can be misleading, especially with imbalanced data.
• The confusion matrix provides a detailed breakdown of a model's predictions into True/False Positives and Negatives.
• Precision and Recall are powerful metrics that measure different aspects of a model's performance, and there is often a trade-off between them.
• The F1-Score provides a balanced measure of both precision and recall.
• You can "tune" a logistic regression model using hyperparameters like penalty and C to control regularization.

Preview of the Next Lesson:
We've now covered a foundational classification algorithm. In the next lesson, we will explore a completely different approach with the k-Nearest Neighbors (KNN) algorithm. Unlike logistic regression, which learns a mathematical function to separate the data, KNN is a non-parametric method that makes predictions based on the principle of "similarity." This will broaden your toolkit and show you that there are often multiple, distinct ways to solve the same problem.