Interpreting Logistic Regression Coefficients

From Probability to a Linear Relationship

As we discussed, the relationship between a feature $x_j$ and the final probability $P(y=1)$ is not linear; it's an S-shaped curve. This makes interpreting the coefficient ${\beta }_j$ tricky. We can't say "a one-unit increase in $x_j$ adds ${\beta }_j$ to the probability."

To find a linear relationship that we can interpret, we need to transform the y-axis. This involves two concepts you may have heard of but might not have used much in practice: odds and log-odds.

Let's build a solid foundation for these terms.

As the video explained, we have a chain of transformations:

1. Probability (P): The chance of an event happening, ranging from 0 to 1.
2. Odds: The ratio of the probability of an event happening to it not happening. Calculated as $\frac{P}{1-P}$, it ranges from 0 to $\infty$.
3. Log-Odds: The natural logarithm of the odds, $\ln \left(\frac{P}{1-P}\right)$. This transformation is key because it maps probabilities onto an unbounded, symmetric scale from $-\infty$ to $+\infty$, just like the output of a linear model. This log-odds function is also known as the logit function.

The Logit Link: Connecting to the Linear Model

Now for the "aha!" moment. Let's revisit the logistic regression equation from last lesson and do a little algebra.
We started with:
$P(y=1)=\frac{1}{1+e^{-z}}$
where $z={\beta }_0+{\beta }_1{x}_1+\cdots +{\beta }_p{x}_p$.

If we rearrange this equation to solve for $z$, we find:
$\ln \left(\frac{P}{1-P}\right)=z$
This reveals something profound: logistic regression is a linear model for the log-odds of the outcome.

The S-shaped curve we see when plotting probabilities becomes a straight line when we plot the log-odds instead. This is why we can use a linear equation ($z$) to model a non-linear probability.

This image perfectly illustrates the transformation.

So, a one-unit increase in a feature $x_j$ results in an increase of ${\beta }_j$ in the log-odds of the outcome. This is mathematically correct, but "log-odds" are not very intuitive to explain. We need one more step to make our interpretation practical.

Interpreting Coefficients as Odds Ratios

Since the model is linear in the log-odds, we can compare the log-odds for an observation with feature value $x_j$ to one with feature value $x_j+1$.

The change in log-odds is:
$\text{log-odds}(x_j+1)-\text{log-odds}(x_j)={\beta }_j$

Using the logarithm rule $\ln (A)-\ln (B)=\ln (A/B)$, we get:
$\ln \left(\frac{\text{odds}(x_j+1)}{\text{odds}(x_j)}\right)={\beta }_j$

To get rid of the log, we exponentiate both sides:
$\frac{\text{odds}(x_j+1)}{\text{odds}(x_j)}=e^{{\beta }_j}$

This final expression, $e^{{\beta }_j}$, is the odds ratio. It tells us how the odds of the outcome are multiplied when the feature $x_j$ increases by one unit. This is the most common way to interpret logistic regression coefficients.

• If ${\beta }_j>0$, then $e^{{\beta }_j}>1$, and a one-unit increase in $x_j$ increases the odds.
• If ${\beta }_j<0$, then $e^{{\beta }_j}<1$, and a one-unit increase in $x_j$ decreases the odds.
• If ${\beta }_j=0$, then $e^{{\beta }_j}=1$, and the feature $x_j$ has no effect on the odds.

Let's see this in action with some clear examples.

To solidify this, let's look at a textual explanation and a concrete example.

Summary of Interpretation Rules:

• For a continuous feature: "A one-unit increase in [feature name] is associated with the odds of [outcome = 1] being multiplied by a factor of $e^{{\beta }_j}$, holding all other features constant."
  • If $e^{{\beta }_j}=1.15$, you can also say the odds increase by 15%.
  • If $e^{{\beta }_j}=0.80$, you can also say the odds decrease by 20%.
• For a categorical feature: "The odds of [outcome = 1] for [category name] are $e^{{\beta }_j}$ times the odds for the [reference category], holding all other features constant."

Now, let's test your understanding with a practical exercise.

Conclusion

Fantastic work! This lesson took you deep into the heart of interpreting logistic regression, a cornerstone of explainable machine learning. You've moved beyond simply generating a prediction to understanding how the model arrives at that prediction.

Key Takeaways:

• Logistic regression models a linear relationship with the log-odds of the outcome, not the probability itself.
• The log-odds are also known as the logit function: $\ln (P/(1-P))$.
• A coefficient ${\beta }_j$ represents the change in the log-odds for a one-unit change in feature $x_j$.
• For a more intuitive interpretation, we calculate the odds ratio by exponentiating the coefficient: $e^{{\beta }_j}$.
• This odds ratio tells us the multiplicative factor by which the odds of the outcome change for a one-unit increase in the feature.

Preview of the Next Lesson:
We've mastered binary (two-class) classification. But what if we need to classify an outcome into three or more categories, like identifying a handwritten digit (0-9) or classifying news articles into topics (sports, politics, technology)? In our next lesson, you will learn how to extend logistic regression to multi-class problems using the softmax function. This will open up a much wider range of classification problems for you to solve.