Statistical Tests for Data Adherence

The Foundation: Standardised Deviations

Before we can test anything, we need a way to measure the "error" or deviation at each age. We don't just look at the raw difference between observed deaths and the deaths predicted by our graduation. Instead, we standardize this difference by its expected standard deviation. This gives us the standardised deviation, $z_x$, for each age $x$.

Under the null hypothesis ($H_0$) that our graduated rates (${\hat{\mu }}_x$ or ${\hat{q}}_x$) are the true, underlying rates for the population, the number of observed deaths, $d_x$, can be modeled. For instance, using a Poisson model, $d_x$ is a realization of a random variable $D_x\sim \text{Poisson}(E_x^c{\hat{\mu }}_x)$, where $E_x^c$ is the central exposed to risk. The mean and variance are both $E_x^c{\hat{\mu }}_x$.

This allows us to construct a standardised test statistic for each age:

For a large enough number of expected deaths, the Central Limit Theorem implies that this $Z_x$ will be approximately a standard normal random variable, $Z_x\sim N(0,1)$. The calculated value for our specific data is denoted $z_x$. This $z_x$ value is the fundamental building block for the tests we will explore.

The following document formally defines this statistic for different underlying models.

The Chi-Squared Test: An Overall Measure of Fit

The Chi-squared test provides an aggregate measure of adherence across all ages. Since you know that each $z_x$ is approximately a standard normal variate, what happens when we square them and add them up? You've got it: the result is a Chi-squared distributed variable.

The test proceeds as follows:

1. Null Hypothesis ($H_0$): The graduated rates are the true underlying rates for the population (i.e., the graduation provides a good fit).
2. Alternative Hypothesis ($H_A$): The graduated rates are not the true rates.
3. Test Statistic: We calculate the statistic $X={\sum }_x{z}_x^2$.
4. Distribution under $H_0$: $X\sim {\chi }_k^2$, where $k$ is the degrees of freedom.
5. Degrees of Freedom ($k$): This is a crucial step. The degrees of freedom are the number of age groups being tested ($m$) minus the number of parameters ($p$) that were estimated to produce the graduation.
   • For a parametric graduation (e.g., Gompertz with 2 parameters), $k=m-2$.
   • For a spline graduation, $p$ would be the number of spline parameters or effective degrees of freedom.
6. Decision Rule: We compare our calculated $X$ to the critical value from the ${\chi }_k^2$ distribution at a chosen significance level (e.g., 5%). Since we are squaring deviations, only large positive values of $X$ suggest a poor fit, making this an upper-tailed test. If $X>{\chi }_{k,0.95}^2$, we reject $H_0$.

The following video gives a great general introduction to the Chi-squared goodness-of-fit test.

Now, let's see this applied in an actuarial context. The following resource shows a worked example from an actual exam paper.

Disadvantages of the Chi-Squared Test

While powerful, the Chi-squared test is not perfect. It gives a single summary number, which can hide important details. The document from Oxford University you reviewed earlier also lists these drawbacks.

This second point—insensitivity to bias—is what motivates the tests we will cover in the next lesson.

The Standardised Deviations Test

To address the first disadvantage of the Chi-squared test (that individual large deviations might be overlooked), we can use the Standardised Deviations test.

The logic is elegant: if the $z_x$ values are truly from a standard normal distribution, then their own distribution should look like it. We can test this formally using... another Chi-squared goodness-of-fit test!

The procedure is:

1. Calculate the $z_x$ for all $m$ age groups.
2. Divide the range of possible $z_x$ values into several bins (e.g., $z<-2$, $-2\le z<-1$, $-1\le z<0$, etc.).
3. For each bin, count the observed number of $z_x$ values that fall into it.
4. For each bin, calculate the expected number of $z_x$ values. This is $m\times p_i$, where $p_i$ is the probability of a standard normal variable falling into that bin. For example, the probability for the bin $(-1,0)$ is approximately 0.3413.
5. Calculate the Pearson's Chi-squared statistic: $X={\sum }_{\text{bins}}\frac{(\text{Observed}-\text{Expected}{)}^2}{\text{Expected}}$.
6. Compare this statistic to a ${\chi }^2$ distribution with (number of bins - 1) degrees of freedom.

Let's see the formal description of this test.

Conclusion

In this lesson, we have equipped ourselves with the primary tools for checking whether a graduation adheres to the observed data. This is the first and most important step in assessing the quality of our work.

Key Takeaways:

• Standardised Deviation ($z_x$): The foundational unit for adherence tests. It measures the difference between observed and expected deaths in units of standard deviation. Under the null hypothesis, it follows an $N(0,1)$ distribution.
• Chi-Squared Test: An aggregate test that sums the squares of the standardised deviations ($\sum z_x^2$). It tests for the overall size of deviations across all ages. A large value indicates a poor fit.
• Degrees of Freedom: The calculation for the Chi-squared test must account for the number of parameters estimated in the graduation: $k=m-p$.
• Standardised Deviations Test: A more granular test that checks if the distribution of the $z_x$ values themselves conforms to a standard normal distribution. It does this by grouping the $z_x$ values into bins and performing a Pearson's Chi-squared test.

Preview of the Next Lesson

While the Chi-squared and Standardised Deviations tests are excellent for assessing the magnitude of errors, they are not designed to detect systematic patterns. For example, a graduation could have small deviations that are consistently positive, indicating a bias that these tests would miss. In our next lesson, we will explore a suite of tests designed specifically to detect such patterns, including the Signs Test, the Grouping of Signs Test, and tests for Serial Correlations.