Likelihood Ratio Test for Nested GLMs

Hello! Welcome to the seventh lesson in our intensive midterm prep course.

In our last lesson, you learned how to compute and interpret deviance as a measure of goodness-of-fit for a single GLM. This gave us a tool to answer the question, "How well does my model fit the data?"

Now, we'll build directly on that knowledge to answer a different, but equally important, question: "Is my model with more predictors statistically better than a simpler version?" This lesson focuses on the Likelihood Ratio Test (LRT), a fundamental method for comparing models.

Your learning outcome is to be able to perform and interpret a Likelihood Ratio Test (LRT) to compare nested GLMs using their deviances. This is a common exam task where you might be asked whether a group of variables can be dropped from a model.

1. The Core Idea: Comparing Nested Models

The Likelihood Ratio Test is designed for a specific situation: comparing two nested models.

Full Model (or Unrestricted Model): The more complex model, containing a larger set of predictors.
Reduced Model (or Restricted Model): A simpler model where one or more parameters from the full model have been constrained, usually by setting their coefficients to zero (i.e., removing the predictors).

For example, if your full model is admit ~ gre + gpa + rank, a nested reduced model could be admit ~ gre + gpa. The reduced model is a special case of the full model. You cannot use an LRT to compare non-nested models like admit ~ gre + gpa and admit ~ gre + rank.

The LRT helps us decide if the extra complexity of the full model is justified by a significantly better fit to the data.

Hypotheses for the LRT:

Null Hypothesis ( $H_{0}$ ): The reduced model is adequate. The extra parameters in the full model are all equal to zero and do not improve the model fit.
Alternative Hypothesis ( $H_{A}$ ): The full model provides a statistically significant better fit than the reduced model.

To build your intuition for what the test is doing, the following video gives a great graphical explanation.

Likelihood ratio test - introduction

This video from Ben Lambert provides the intuition behind the LRT. It visually demonstrates how the test evaluates whether the increase in the log-likelihood function gained by moving from the restricted (null) model to the unrestricted (full) model is large enough to be considered significant.

Watch from 01:24 to 04:24. Focus on the graph of the log-likelihood function. Notice how the test is essentially measuring the vertical distance between the log-likelihood under the null hypothesis (L of H₀) and the maximum possible log-likelihood (L of H₁). A large distance suggests the null hypothesis is unlikely.

2. The Test Statistic: From Log-Likelihoods to Deviances

As the video explained, the test statistic is based on the difference in the log-likelihoods of the two models. The formal definition of the LRT statistic is:

LRT statistic = - 2 ln (\frac{Likelihood of Reduced Model}{Likelihood of Full Model}) = 2 \times (log-likelihood_{f u l l} - log-likelihood_{r e d u ce d})

This should look familiar! In the previous lesson, we defined the residual deviance of a model as:

D = 2 \times (log-likelihood_{s a t u r a t e d} - log-likelihood_{m o d e l})

Let's use this definition to make a powerful connection.

Deviance of the reduced model: $D_{r e d} = 2 \times (l_{s a t} - l_{r e d})$
Deviance of the full model: $D_{f u l l} = 2 \times (l_{s a t} - l_{f u l l})$

Now, let's calculate the difference between these two deviances:

Δ D = D_{r e d} - D_{f u l l} = [2 \times (l_{s a t} - l_{r e d})] - [2 \times (l_{s a t} - l_{f u l l})]

Δ D = 2 \times l_{s a t} - 2 \times l_{r e d} - 2 \times l_{s a t} + 2 \times l_{f u l l}

Δ D = 2 \times (l_{f u l l} - l_{r e d})

This is our key insight: The Likelihood Ratio Test statistic is simply the difference in the residual deviances of the two nested models.

Under the null hypothesis, this test statistic $Δ D$ follows a chi-squared ( $χ^{2}$ ) distribution. The degrees of freedom for this test is the difference in the number of parameters between the two models, which is simply the number of predictors you removed to create the reduced model.

3. How to Perform an LRT: A Step-by-Step Guide

Here is a clear, algorithmic approach for your exam:

Define Models: Identify the full model and the reduced model. Confirm they are nested.
Fit Models & Find Deviance: Obtain the residual deviance and the residual degrees of freedom for both models. Let's call them $D_{f u l l}$ , $d f_{f u l l}$ , $D_{r e d}$ , and $d f_{r e d}$ .
Calculate the Test Statistic ( $Δ D$ ): $Δ D = D_{r e d} - D_{f u l l}$
Calculate the Degrees of Freedom ( $Δ df$ ): $Δ df = d f_{r e d} - d f_{f u l l} = (number of predictors in full model) - (number of predictors in reduced model)$
Find the p-value: Compare the test statistic $Δ D$ to a $χ^{2}$ distribution with $Δ df$ degrees of freedom.
Draw a Conclusion:
- If the p-value is small (e.g., $p < 0.05$ ), reject $H_{0}$ . Conclude that the full model is a significantly better fit. The variables you removed are statistically important.
- If the p-value is large, fail to reject $H_{0}$ . Conclude there is not enough evidence that the full model is better. You can use the simpler (reduced) model for parsimony.

4. A Worked Example

Let's walk through an example of testing whether the variables math and science significantly improve a logistic regression model that already contains female and read.

How are the likelihood ratio, Wald, and Lagrange multiplier (score) tests different and/or similar?

This article from the UCLA Statistical Consulting Group provides a perfect, practical example of performing an LRT. It shows the output for both the full and reduced models and then calculates the test statistic by hand.

Read the section titled 'The likelihood ratio test'. It begins by stating the formula and then walks through a logistic regression example, showing the output for a reduced model and a full model. Follow the calculation to see how the log-likelihoods are used.

Let's summarize the results from that example to reinforce the steps:

Full Model: hiwrite ~ female + read + math + science. Log-likelihood = -84.42.
Reduced Model: hiwrite ~ female + read. Log-likelihood = -102.45.
Test Statistic: $Δ D = 2 \times (- 84.42 - (- 102.45)) = 2 \times 18.03 = 36.06$ .
Degrees of Freedom: The full model has 4 predictors, the reduced has 2. So, $Δ df = 4 - 2 = 2$ .
Conclusion: The test statistic of 36.06 is extremely large for a $χ^{2}$ distribution with 2 degrees of freedom. The p-value is less than 0.001. We reject the null hypothesis and conclude that adding math and science significantly improves the model.

Interpreting Software Output

In an exam or in practice, you won't usually calculate this by hand. You'll interpret software output. Statistical packages often provide an "Analysis of Deviance" table that does the LRT for you.

Analysis of Deviance Table for GLM Comparison — This R output shows a direct comparison of two nested models using an LRT. Model 1 is the reduced model and Model 2 is the full model.

Let's break down this table, which is the kind of output you can expect to see:

Model 1 (Reduced): Resid. Dev = 45.454, Resid. Df = 39.
Model 2 (Full): Resid. Dev = 45.062, Resid. Df = 37.
Df: This is $Δ df = 39 - 37 = 2$ . It's the number of parameters added to the full model.
Deviance: This is the test statistic $Δ D = 45.454 - 45.062 = 0.392$ .
Pr(>Chi): This is the p-value, 0.822. Since this p-value is very large, we fail to reject the null hypothesis. We conclude that the additional predictors in Model 2 do not significantly improve the fit, so we would prefer the simpler Model 1.

Conclusion

Today you've added a powerful tool to your statistical arsenal. The Likelihood Ratio Test provides a principled way to decide if adding complexity to a model is statistically justified.

Key Takeaways:

The LRT is used to compare nested models.
The null hypothesis ( $H_{0}$ ) states that the simpler (reduced) model is sufficient.
The test statistic is the difference in the residual deviances of the two models: $Δ D = D_{r e d u ce d} - D_{f u l l}$ .
This statistic follows a $χ^{2}$ distribution with degrees of freedom equal to the number of parameters excluded from the reduced model.
A small p-value means the full model is significantly better; a large p-value means you can stick with the simpler model.

The LRT is a robust and widely used test, but it requires you to fit two separate models. In our next lesson, we will explore the Wald test, which allows you to test the significance of individual model coefficients using the output from just a single model.