Significance Testing in Logistic Regression

1. A Quick Refresher: The Three Big Tests

In GLMs, there are three main tests for assessing model parameters: the Wald test, the Likelihood Ratio Test (LRT), and the Score test. All three are based on the log-likelihood function, but they measure "extremeness" in different ways. For your exam, the Wald and Likelihood Ratio tests are the most important.

To get a quick, high-level intuition for how these tests relate to each other, please watch this short introductory video.

Now, let's dive into the practical application of the two most common tests: the Wald and the Likelihood Ratio Test.

2. The Wald Test: Significance of a Single Predictor

The Wald test is the most direct way to test the significance of a single predictor. It answers the question: Is the coefficient ${\beta }_i$ for this predictor significantly different from zero?

• Null Hypothesis ($H_0$): ${\beta }_i=0$ (The predictor has no effect on the log-odds of the outcome).
• Alternative Hypothesis ($H_A$): ${\beta }_i\ne 0$ (The predictor has an effect).

The test statistic is very simple and should feel familiar. It is the estimated coefficient divided by its standard error.

This $Z$-statistic is then compared to a standard normal distribution to get a p-value. In most software output, this is done for you automatically.

How to Use the Wald Test in Practice

For your exam, you simply need to locate the correct line in the regression output table and read the p-value.

As you can see from that reading, you look for the P-Value (sometimes labeled P>|z| or Sig.). If this value is less than your chosen significance level (e.g., $\alpha =0.05$), you reject the null hypothesis and conclude the predictor is statistically significant.

This video from StatQuest provides an excellent walkthrough of interpreting this exact part of the output for both continuous and categorical predictors.

Key Takeaway for the Wald Test: It's your go-to method for quickly assessing the significance of a single predictor directly from the main coefficient table in your regression output.

3. The Likelihood Ratio Test (LRT): Comparing Nested Models

The Likelihood Ratio Test (LRT) is a more general and often more reliable method. Instead of testing a single coefficient, it compares the fit of two nested models.

• Full Model: The more complex model, which includes the predictor(s) you want to test.
• Reduced Model: The simpler model, where the coefficient(s) for the predictor(s) you are testing are constrained to be zero (i.e., they are removed from the model).

The LRT answers the question: Does the full model fit the data significantly better than the reduced model?

• Null Hypothesis ($H_0$): The reduced model is sufficient (i.e., the predictor(s) we removed have no effect, ${\beta }_i=\cdots ={\beta }_k=0$).
• Alternative Hypothesis ($H_A$): The full model provides a significantly better fit (at least one of the removed predictors has an effect).

The test statistic, often called $G^2$, is calculated from the log-likelihoods (or deviances) of the two models:

This can also be written as:

This $G^2$ statistic follows a chi-square (${\chi }^2$) distribution with degrees of freedom equal to the number of parameters you removed to create the reduced model.

How to Use the LRT in Practice

To perform an LRT, you need the log-likelihood or deviance from both the full and reduced models.

This first resource explains the mechanics and shows how to interpret a "Deviance Table," which is a common way software presents LRT results.

Sometimes, you may need to calculate the LRT statistic by hand from the output of two separate models. This next resource provides a clear, step-by-step example.

Key Takeaway for the LRT: It is the standard method for testing the significance of a group of predictors (e.g., a categorical variable with 3+ levels, or a set of interaction terms) or for formally comparing any two nested models.

4. Exam Strategy: Wald vs. LRT

For your exam, you need to know which test to use and when. Here is a simple decision guide.

While the tests are computationally different, for large samples, they will almost always lead to the same conclusion. The LRT is generally considered statistically superior, especially in smaller samples or when parameter estimates are large.

This final resource provides a superb summary comparing the tests, including a graphic that visually explains the difference.

Conclusion

You now have two powerful tools to formally assess the significance of predictors in a logistic regression model.

Key Takeaways:

• Wald Test: Used to test the significance of a single coefficient ($H_0:{\beta }_i=0$). The test statistic is $Z={\hat{\beta }}_i/\text{SE}({\hat{\beta }}_i)$. The p-value is typically found in the main regression output table.
• Likelihood Ratio Test (LRT): Used to compare nested models. It's ideal for testing the joint significance of multiple predictors. The test statistic is $G^2={\text{Deviance}}_{\text{reduced}}-{\text{Deviance}}_{\text{full}}$ and follows a ${\chi }^2$ distribution.
• Connection: A significant p-value from either test ($p<0.05$) corresponds to the finding from our last lesson: the 95% confidence interval for the odds ratio does not contain 1.

In our next and final lesson on logistic regression, we'll zoom out from individual predictors and look at the model as a whole. We will explore how to use the Pearson chi-square statistic to assess the overall goodness-of-fit of the model.