Wald, Deviance, and Score Tests for GLM Coefficients

The Unifying Concept: The Log-Likelihood Function

All three tests are different ways of asking the same fundamental question: "How much evidence do we have against the null hypothesis that a coefficient is zero ($H_0:{\beta }_j=0$)?". They each answer this question by examining the log-likelihood function.

Recall that the log-likelihood function, $\ell (\beta )$, measures how probable our observed data is, given a particular value for a model coefficient $\beta$. We find the "best" coefficient estimate, $\hat{\beta }$ (the Maximum Likelihood Estimate or MLE), by finding the value of $\beta$ that maximizes this function.

The three tests can be understood visually as measuring the "discrepancy" between the MLE ($\hat{\beta }$) and the value under the null hypothesis (${\beta }_0$, which is usually 0) in different ways on the log-likelihood curve.

To get a clear overview of this core idea, let's watch a short video.

Now that we have the general intuition, let's examine each test in more detail.

1. The Likelihood Ratio Test (LRT) or Deviance Test

The Likelihood Ratio Test is arguably the most intuitive and robust of the three. It directly uses the concept of deviance that we learned in the previous lesson.

• Core Idea: The LRT asks: "How much worse does our model fit if we force the coefficient ${\beta }_j$ to be zero?" It compares the log-likelihood of the full model (with ${\beta }_j$) to the reduced model (without ${\beta }_j$). A large drop in log-likelihood when we remove the predictor is strong evidence that the predictor is important.

• The Test Statistic: The test statistic is calculated from the log-likelihoods of the two models:

  $$\Lambda =-2\times [\ell (\text{reduced model})-\ell (\text{full model})]$$

  This should look familiar! This is precisely the difference in the deviances between the reduced and full models. This statistic follows a ${\chi }^2$ distribution with degrees of freedom equal to the number of parameters removed (in this case, 1).

• Practical Implementation: You must fit two models: one with the predictor(s) of interest and one without.

Let's see how this is done in practice. The following video demonstrates how to perform an LRT in R to compare nested models.

Pros & Cons of LRT:

• Pro: Generally considered the most reliable of the three tests. It is invariant to re-parameterization (e.g., testing $H_0:{\beta }_j=0$ gives the same result as testing $H_0:e^{{\beta }_j}=1$).
• Con: It is the most computationally demanding, as it requires fitting two separate models.

2. The Wald Test

The Wald test is the most common test you'll see in statistical output, typically presented as a z-statistic or t-statistic next to each coefficient.

• Core Idea: The Wald test asks: "How many standard errors is our estimated coefficient ${\hat{\beta }}_j$ away from zero?" It is based on the assumption that the log-likelihood function is approximately quadratic around the maximum.

• The Test Statistic: The statistic is simple and should be familiar from linear regression:
  $W=\frac{{\hat{\beta }}_j-0}{\text{se}({\hat{\beta }}_j)}$
  This statistic $W$ is compared to a standard Normal distribution (a Z-test). Squaring it, $W^2$, gives a statistic that follows a ${\chi }^2$ distribution with 1 degree of freedom.

• Practical Implementation: You only need to fit the full model. The coefficient estimate (${\hat{\beta }}_j$) and its standard error ($\text{se}({\hat{\beta }}_j)$) are used to compute the test.

The key weakness of the Wald test is its reliance on the quadratic approximation of the log-likelihood function. When this approximation is poor, the Wald test can be unreliable.

Pros & Cons of Wald Test:

• Pro: Computationally cheap (only requires one model) and easy to interpret. It's the default in most software summaries.
• Con: It is not invariant to re-parameterization. It can perform poorly with small sample sizes or when the true parameter value is far from the null, as the quadratic approximation of the log-likelihood may be inaccurate.

3. The Score Test (or Lagrange Multiplier Test)

The Score test takes a different approach, starting from the null hypothesis.

• Core Idea: The Score test asks: "If we were at the null hypothesis value (${\beta }_j=0$), how steep is the 'slope' of the log-likelihood function?" The slope of the log-likelihood is called the score function. A steep slope suggests that the likelihood would increase substantially if we allowed ${\beta }_j$ to be non-zero, providing evidence against the null.

• The Test Statistic: The formula involves the score function and the information matrix, but the key concept is that it evaluates the gradient at the null value.

• Practical Implementation: You only need to fit the reduced model (the model without the predictor of interest).

Pros & Cons of Score Test:

• Pro: Computationally cheap, as it only requires fitting the simpler, reduced model. Like the LRT, it is invariant to re-parameterization. This makes it particularly useful for screening a large number of potential predictors to add to a model (a procedure called forward selection).
• Con: It is generally less powerful than the LRT, especially in smaller samples.

Comparison and Practical Guidance

Now that we've described all three, let's put them together. For large sample sizes, all three tests will give very similar results. However, in finite samples, they can differ.

This final reading provides a fantastic comparison, summarizing what each test does and why you might choose one over another.

Here is a summary table to help consolidate your understanding:

Conclusion

In this lesson, you've learned about the three fundamental tests for assessing the significance of coefficients in a GLM. You now have the functional understanding to not just read statistical output, but to critically evaluate it.

Key Takeaways:

• The Wald, Likelihood Ratio (LRT), and Score tests are three different ways of testing hypotheses about GLM coefficients by examining the log-likelihood function.
• The LRT (Deviance Test) is generally the most robust method. It compares the fit of nested models and its test statistic is the difference in deviance.
• The Wald Test is the most convenient as it is provided in standard output, but its reliance on a quadratic approximation makes it less reliable in small samples.
• The Score Test is computationally efficient, requiring only the null model, which makes it ideal for variable screening procedures.
• While the tests converge with large samples, for any given analysis, the LRT is often the preferred choice for formal hypothesis testing due to its superior statistical properties.

Preview of the Next Lesson:

We have now described the three main tests. In the next lesson, we will build directly on this foundation to compare their practical and theoretical differences in more depth, looking at situations where their results might diverge and what that tells you about your model and data.