Mortality Rate Graduation: Graphics & Splines

Graphical Graduation: The Art of Smoothing

The most intuitive way to graduate crude rates is to simply plot them and draw a smooth curve that you feel best represents the underlying trend. This is the essence of graphical graduation. The process involves:

1. Plotting the crude mortality rates (${\hat{m}}_x$) on a graph, often with a logarithmic scale for the y-axis.
2. Drawing a freehand curve that passes through the "middle" of the plotted points, ignoring what appear to be random fluctuations.
3. Reading the graduated rates directly from the hand-drawn curve at each age.

While this method is simple to understand and can be a useful exploratory tool, it has significant drawbacks:

• Subjectivity: Two different actuaries will almost certainly produce two different sets of graduated rates from the same data.
• Lack of Reproducibility: The process is not based on a formal mathematical or statistical procedure.
• Difficulty in Testing: It is challenging to statistically assess the goodness-of-fit of a hand-drawn curve.

Because of these limitations, graphical graduation is rarely used for final, formal graduations in modern practice. However, it serves as the conceptual foundation for more sophisticated methods that aim to automate this "curve-drawing" process, which leads us to splines.

Spline-Based Graduation: A Rigorous Approach to Flexibility

How can we create a flexible curve that follows the data, but do so in an objective and reproducible way? The answer lies in using splines. A spline is a function constructed by joining several simple polynomial functions together, piece by piece.

The points where the polynomial pieces connect are called knots. To ensure the final curve is smooth, we impose continuity conditions at these knots. For cubic splines, which are very common in mortality work, we require that the value of the function, its first derivative, and its second derivative are all continuous at each knot. This ensures a seamless transition between the polynomial segments, resulting in a curve that is not only continuous but also has continuous slope and curvature.

The following short video provides an excellent introduction to the core concepts of cubic splines.

The great advantage of splines is their flexibility. By placing knots at appropriate ages, we can create a curve that captures complex features in mortality data, such as the "accident hump" in the teenage years, which traditional parametric models like Gompertz or Makeham would miss.

P-splines: Taming Flexibility with Penalization

Splines solve the flexibility problem, but they introduce a new one: how many knots should we use, and where should we place them?

• Too few knots, and the spline might not be flexible enough (high bias).
• Too many knots, and the spline can become overly "wiggly," fitting to random noise in the data instead of the underlying trend (high variance).

This is a classic bias-variance trade-off. Penalized splines, or P-splines, offer an elegant solution. The idea, introduced by Eilers and Marx (1996), is to use a generous number of knots (often regularly spaced) and then add a penalty to the estimation process that discourages "roughness."

The estimation is typically done by maximizing a penalized log-likelihood:

Let's break this down:

• $l(\theta )$ is the standard log-likelihood function (e.g., from a Poisson model), which encourages the curve to fit the data closely.
• $P(\theta )$ is the roughness penalty. It's a function of the spline's coefficients ($\theta$) that becomes large if the curve is too wiggly. A common choice for $P(\theta )$ is the sum of the squared second differences of adjacent spline coefficients. A smooth, straight line would have a penalty of zero.
• $\lambda$ is the smoothing parameter. This non-negative parameter controls the trade-off.
  • If $\lambda =0$, there is no penalty, and the spline will fit the data as closely as possible, likely overfitting.
  • As $\lambda \to \infty$, the penalty dominates, forcing the spline to become as smooth as possible. For a second-difference penalty, this forces the fit towards a straight line on the log-mortality scale (i.e., a Gompertz model).

This concept of penalized estimation is central to modern statistics (e.g., ridge regression, LASSO) and provides a powerful way to manage model complexity. The following video from Stanford's Statistical Learning course explains this fundamental idea beautifully.

The P-spline Graduation Process

Putting this all together, graduating with P-splines follows a clear, statistical workflow.

The choice of the smoothing parameter $\lambda$ is crucial. Rather than being chosen arbitrarily, it is typically selected using statistical criteria like minimizing the Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), or through cross-validation. This makes the entire process objective and data-driven.

The result is a smooth curve that captures the underlying trend in the data without overfitting to random noise.

A sophisticated concept related to the penalty is the effective degrees of freedom. While a spline model might have many parameters (coefficients), the penalty "uses up" some of these parameters to enforce smoothness. The effective degrees of freedom, which is typically a non-integer value, measures the true flexibility of the fitted model. A model with a large $\lambda$ will have very few effective degrees of freedom. This concept, which is discussed in the optional video segment below, provides a formal way to quantify model complexity in penalized models.

Conclusion

In this lesson, we have moved from the subjective art of graphical graduation to the powerful and objective science of spline-based methods. P-splines, in particular, represent the state-of-the-art in non-parametric graduation, offering a superb balance between fidelity to the data and the necessary smoothness of a mortality curve.

Key Takeaways:

• Graphical Graduation: An intuitive but subjective method involving hand-drawing a smooth curve through plotted crude rates.
• Cubic Splines: Provide a flexible way to model mortality curves by joining cubic polynomial pieces smoothly. They can capture complex features that parametric models cannot.
• P-splines (Penalized Splines): Address the knot-selection problem by using many knots and adding a roughness penalty to the log-likelihood.
• The Smoothing Parameter ($\lambda$): This crucial parameter controls the trade-off between fitting the data closely (adherence) and keeping the curve from being too wiggly (smoothness). It is chosen objectively using statistical criteria like AIC, BIC, or cross-validation.
• The Process: P-spline graduation is a complete statistical modeling procedure that combines a statistical model for deaths (e.g., Poisson), a flexible spline function, and a penalty term to deliver a robust and reproducible graduation.

Preview of the Next Lesson

We have now learned three distinct ways to graduate mortality rates: using parametric formulas, referencing a standard table, and using flexible splines. But how do we judge the quality of our results? Is the fit good enough? Is the curve sufficiently smooth? In our next lesson, we will dive into the essential statistical tests used to assess a graduation, covering tests for adherence to data, such as the Chi-squared test.