Logistic Regression: Training and Interpretation

Introduction

In our last two lessons, we assembled the conceptual building blocks of logistic regression. We saw how it uses a linear model combined with a sigmoid function to output a probability, and how it learns by minimizing cross-entropy loss.

Today, we put it all together. We will move from theory to practice by training a logistic regression model using your preferred library, scikit-learn. More importantly, you will learn how to look "under the hood" of the trained model to interpret its coefficients. This is a crucial skill for any data scientist, as it allows you to explain why your model makes the predictions it does.

By the end of this lesson, you will be able to train a logistic regression classifier and confidently interpret its coefficients in terms of log-odds.

Training a Logistic Regression Model with Scikit-Learn

Let's start by seeing the model in action. The process follows the standard, elegant workflow you've seen with other scikit-learn estimators: instantiate the model, fit it to the training data, and predict on new data.

The following video provides a clear, practical demonstration. It uses a simple dataset to predict whether a person will buy insurance based on their age.

As you saw, the core of the implementation is remarkably concise:

1. Import and Instantiate:

   from sklearn.linear_model import LogisticRegression
   model = LogisticRegression()
   

2. Train the Model:

   model.fit(X_train, y_train)
   

3. Make Predictions:

   predictions = model.predict(X_test)
   probabilities = model.predict_proba(X_test)
   

The fit method handles all the complex optimization work of minimizing the cross-entropy loss function we discussed in the last lesson. The result is a fitted model—a sigmoid curve—that best separates the two classes.

Interpreting Model Coefficients: From Predictions to Understanding

Making predictions is great, but understanding how the model arrives at them is even better. With linear models like logistic regression, we can inspect the learned coefficients to understand the relationship between each feature and the outcome.

However, unlike in linear regression where the coefficient directly tells you the change in the output value, in logistic regression, the interpretation is a bit more nuanced.

The Concept of Log-Odds

Recall that logistic regression models the probability $p$ of an outcome using the sigmoid function:
$p=\frac{1}{1+e^{-z}}$
where $z$ is a linear combination of the features: $z={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_p{x}_p$.

If we rearrange this equation, we find what the model is actually learning a linear relationship for:

$\log \left(\frac{p}{1-p}\right)={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_p{x}_p$

The term on the left, $\log (p/(1-p))$, is called the log-odds or logit. Logistic regression is a linear model for the log-odds.

This next video from StatQuest provides an excellent visual intuition for this idea. It shows how transforming the y-axis from probability to log-odds turns the S-shaped sigmoid curve into a simple straight line.

What a Coefficient Means

Now that we know the model is linear in the log-odds scale, the interpretation becomes clear:

A coefficient ${\beta }_i$ represents the change in the log-odds of the outcome for a one-unit increase in the feature $x_i$, holding all other features constant.

• A positive coefficient means that as the feature increases, the log-odds of the event occurring increase, making the event more likely.
• A negative coefficient means that as the feature increases, the log-odds of the event occurring decrease, making the event less likely.

To solidify this concept, please read the following brief guide.

From Log-Odds to Interpretable Odds Ratios

While mathematically convenient, "log-odds" aren't very intuitive for humans. We can make the interpretation more concrete by exponentiating the coefficient to get the odds ratio.

The odds ratio is $e^{{\beta }_i}$. It tells you the multiplicative factor by which the odds of the outcome change for a one-unit increase in the feature.

Let's look at a practical example. After you train a model in scikit-learn:

import numpy as np




# Assuming 'model' is a trained LogisticRegression model
# and feature_names is a list of your column names
coefficients = model.coef_[0]
intercept = model.intercept_[0]

for feature, coef in zip(feature_names, coefficients):
    odds_ratio = np.exp(coef)
    print(f"Feature: {feature}")
    print(f"  - Coefficient (log-odds): {coef:.4f}")
    print(f"  - Odds Ratio: {odds_ratio:.4f}")

How to interpret the odds ratio:

• Odds Ratio > 1: The odds of the positive outcome increase. If the odds ratio is 1.4, a one-unit increase in the feature multiplies the odds by 1.4 (a 40% increase in odds).
• Odds Ratio < 1: The odds of the positive outcome decrease. If the odds ratio is 0.7, a one-unit increase in the feature multiplies the odds by 0.7 (a 30% decrease in odds).
• Odds Ratio = 1: The feature has no effect on the odds.

This article provides a perfect, concise code example and explanation of this exact concept.

Conclusion

In this lesson, you bridged the gap between the theory of logistic regression and its practical application. You saw how easy it is to train a model with scikit-learn and, more importantly, learned the correct way to interpret its output.

Key Takeaways:

• Training a LogisticRegression model in scikit-learn uses the familiar .fit() and .predict() methods.
• Logistic regression models a linear relationship with the log-odds of the outcome, not the probability itself.
• A model's coefficient represents the change in the log-odds for a one-unit increase in the corresponding feature.
• To make this more intuitive, you can exponentiate a coefficient ($e^{{\beta }_i}$) to get the odds ratio, which tells you the multiplicative effect of the feature on the outcome's odds.

Preview of the Next Lesson

We've now thoroughly explored our first probabilistic classifier. Next, we will switch gears and look at a completely different, but equally powerful, approach to linear classification: Support Vector Machines (SVMs). Instead of modeling probabilities, SVMs think geometrically, aiming to find the "best" separating line or hyperplane that has the maximum possible margin between classes.