Deviance of GLMs: Calculation and Interpretation

1. What is Deviance?

In linear regression, we measure how well a model fits by summing the squared differences between the observed values ($y_i$) and the fitted values (${\hat{y}}_i$). In GLMs, we use a more general concept based on likelihoods.

Deviance compares the log-likelihood of your fitted model to the log-likelihood of a "perfect" model, called the saturated model.

Here are the three models we use as benchmarks:

1. Saturated Model: A model that fits the data perfectly. It has as many parameters as there are data points, so each fitted value ${\tilde{\mu }}_i$ is exactly equal to the observed value $y_i$. This represents the best possible fit. Let's call its log-likelihood $l(\tilde{\mathbf{\mu }};\mathbf{y})$.
2. Proposed Model: This is your GLM, with $p$ parameters (${\beta }_0,{\beta }_1,\dots$). It generates fitted values $\hat{\mathbf{\mu }}$. Its maximized log-likelihood is $l(\hat{\mathbf{\mu }};\mathbf{y})$.
3. Null Model: The simplest possible model, containing only an intercept (${\beta }_0$). It assumes the predictors have no effect. Its maximized log-likelihood is $l({\hat{\mathbf{\mu }}}_0;\mathbf{y})$.

Using these, we define two key quantities:

• Residual Deviance: This measures the "badness-of-fit" of your proposed model compared to the perfect saturated model.

  $$D(\mathbf{y},\hat{\mathbf{\mu }})=2\left[l(\tilde{\mathbf{\mu }};\mathbf{y})-l(\hat{\mathbf{\mu }};\mathbf{y})\right]$$

  A smaller residual deviance means your model is closer to a perfect fit.

• Null Deviance: This measures the "badness-of-fit" of the intercept-only null model.

  $$D(\mathbf{y},{\hat{\mathbf{\mu }}}_0)=2\left[l(\tilde{\mathbf{\mu }};\mathbf{y})-l({\hat{\mathbf{\mu }}}_0;\mathbf{y})\right]$$

  The null deviance serves as a baseline for how well you can predict the response without using any predictors.

2. Calculating Deviance: A Concrete Example

The definitions above can seem abstract. The best way to understand them is to see them calculated step-by-step. The following video provides an excellent walkthrough for a simple Poisson regression model.

As you saw in the video, the process involves:

1. For each model (proposed, null, saturated), determine the mean prediction ${\mu }_i$ for each data point $y_i$.
2. Use the probability mass/density function (e.g., Poisson PMF) to find the probability of observing each $y_i$ given its corresponding ${\mu }_i$.
3. Sum the logarithms of these probabilities to get the log-likelihood for that model.
4. Finally, plug the log-likelihoods into the deviance formulas.

3. Interpreting Deviance as a Measure of Fit

Now that we can compute deviance, how do we use it?

Analogy to R-squared

The null deviance represents the total variation in the data (like Total Sum of Squares in linear regression), and the residual deviance represents the unexplained variation (like Residual Sum of Squares).

We can construct a "pseudo R-squared" to describe the proportion of deviance explained by the predictors in our model:

A higher $R_D^2$ suggests your model's predictors are explaining more of the variation in the response.

Goodness-of-Fit Test

The residual deviance can be used to formally test the goodness-of-fit of your model.

• Null Hypothesis ($H_0$): Your proposed model fits the data adequately.
• Test Statistic: The residual deviance, $D$.
• Distribution: If $H_0$ is true and the sample size is large enough, $D$ follows a chi-squared distribution with $n-p$ degrees of freedom, where $n$ is the number of observations and $p$ is the number of parameters in your model.

You calculate the deviance, find the p-value from the ${\chi }_{n-p}^2$ distribution, and if $p<\alpha$ (e.g., 0.05), you reject $H_0$. This would mean there is significant evidence that your model does not fit the data well.

4. Practical Shortcuts and Considerations

Manually calculating log-likelihoods is tedious. For exams and real-world analysis, you'll rely on simplified formulas and software output.

Deviance Formulas for Common Distributions

The general deviance formula simplifies for specific distributions. Memorizing these can be a huge advantage on an exam.

Here's a summary of those key formulas, where $D={\sum }_i{d}_i^2$:

Reading Software Output

In practice, you'll get deviance values from software output. It's crucial to know what you're looking at. The slides from LINK show several examples. Notice in the R output on slide 25 (Gamma Log GLM–Intercept and Slope) how the summary includes:

• Null deviance: 252.05 on 199 degrees of freedom
• Residual deviance: 206.47 on 198 degrees of freedom

This tells you the deviance of the intercept-only model and your fitted model, along with their respective degrees of freedom.

Scaled vs. Unscaled Deviance

There's one final, critical detail: the dispersion parameter $\varphi$.

• For Poisson and Binomial models, $\varphi$ is fixed at 1. The deviance we've calculated is what we use.
• For Normal, Gamma, and other distributions, $\varphi$ is not 1 and must be estimated from the data.

This leads to two types of deviance:

• Unscaled Deviance: The value $D$ calculated from the formulas above. This is what R and other software packages typically report as "deviance."
• Scaled Deviance: $D^{\ast }=D/\varphi$. This is the quantity that is actually used for hypothesis testing (i.e., the value you compare to a ${\chi }^2$ distribution).

Conclusion

In this lesson, we demystified deviance, the primary tool for assessing fit in GLMs. You learned how to compute it from log-likelihoods and how to interpret it for model assessment.

Key Takeaways:

• Deviance measures the discrepancy between your model and a perfect (saturated) model. Lower is better.
• Residual Deviance measures your model's fit, while Null Deviance provides a baseline for a model with no predictors.
• The reduction in deviance from null to residual acts like an $R^2$ statistic, showing the explanatory power of your predictors.
• For a goodness-of-fit test, the scaled deviance ($D/\varphi$) is compared to a ${\chi }^2$ distribution with $n-p$ degrees of freedom.
• Remember that for Poisson and Binomial models, $\varphi =1$, so scaled and unscaled deviance are the same.

You've learned how to evaluate a single model. But what if you have two different models and want to know which one is statistically better? In our next lesson, we will use deviance to do just that by performing a Likelihood Ratio Test (LRT) to compare nested GLMs.