Introduction
In our previous lesson, we established a framework for choosing between deterministic and stochastic models, a foundational decision in actuarial work. But once a model is built, how can we be sure it is fit for its purpose? How do we gain confidence in its outputs, especially when they inform multi-million-pound decisions? This brings us to the critical process of model validation.
This lesson will guide you through the structured process of model validation, an essential practice for any actuary. We'll move beyond simple bug-checking to explore a comprehensive framework for assessing a model's integrity. We will pay special attention to two cornerstone quantitative techniques: back-testing and sensitivity testing. Your strong background in probability and statistics will provide an excellent foundation for understanding the principles behind these methods. Our goal is to translate that theoretical knowledge into the practical, risk-managed approach required by the CP1 syllabus.
What is Model Validation?
The great statistician George Box famously stated, "All models are wrong, but some are useful." The purpose of model validation is to understand how a model is wrong, to quantify the associated risks, and to ultimately determine if it is useful and reliable enough for its intended application.
Model validation is an independent and thorough review of a model to assess its accuracy, conceptual soundness, and fitness for purpose. It is the core process for managing model risk—the risk of adverse consequences, such as financial loss or poor decision-making, arising from using an incorrect or misused model.
The validation process can be broken down into several stages, often divided into qualitative and quantitative checks. This flowchart gives a high-level overview of how these components fit together.
To provide a structured way of thinking about model risk, we can break it down into five key sub-categories.
[PDF] ECONOMIC CAPITAL MODEL VALIDATION
This paper from the Swiss Association of Actuaries provides a robust framework for understanding model risk. It defines five distinct sub-categories that form the basis of a structured validation process.
First, read the abstract on page 1, from the start of the paper to the end of the first section. This will introduce the five categories of model risk. Then, turn to page 7 and read the section titled What: assessing model risk. This section provides clear definitions for conceptual, implementation, input, output, and reporting risk. Pay attention to the distinction between output risk (the model's raw results) and reporting risk (how those results are communicated).
As the paper outlines, a thorough validation must address every one of these risk categories:
- Conceptual Risk: Is the underlying theory appropriate for the problem?
- Implementation Risk: Is the model built correctly according to the theory?
- Input Risk: Is the data going into the model accurate and appropriate?
- Output Risk: Are the model's results reasonable and stable?
- Reporting Risk: Are the results communicated clearly and without misrepresentation?
The Validation Process in Practice
Now let's walk through how these risk categories are assessed in a typical validation process. The process combines reviews of documentation and theory with hands-on quantitative testing.
Model Validation: Detailed Process
The video "Model Validation: Detailed Process" by Dimitri Bianco offers a practical walkthrough of the key steps in a validation project. It aligns very closely with the five-part risk framework we just discussed.
Please watch the following segments, which outline the five major components of a validation review: The Five Steps: This gives an overview of the five areas: Data, Model Theory, Model Output, Documentation, and Governance. Data Validation: This covers Input Risk, discussing data sources, exclusions, and sampling. Model Theory: This addresses Conceptual Risk, covering variable selection, model structure, and the testing of statistical assumptions. Model Output: This relates to Output Risk, focusing on accuracy, stability, and robustness. Documentation & Governance: This covers Implementation and Reporting Risk, ensuring the model is documented, repeatable, and has a clear monitoring plan.
The video provides a clear narrative of the validation journey. Let's now zoom in on the two most important quantitative techniques mentioned: sensitivity testing and back-testing.
Quantitative Validation Techniques
While checking the model's theory and documentation is crucial, the core of quantitative validation involves testing the model's behaviour with data.
1. Sensitivity and Stress Testing
Sensitivity testing addresses output risk by asking: "How much do the model's results change when I change an input assumption?" It helps identify the model's most significant drivers.
There are several related techniques:
- Sensitivity Testing: Adjusting a single assumption (e.g., increasing the mortality rate by 1%) to see the impact on the output (e.g., the reserve).
- Scenario Testing: Adjusting multiple assumptions at once to model a plausible future scenario (e.g., a recession involving falling interest rates and rising unemployment).
- Stress Testing: A specific type of scenario test that models a severe but plausible event (e.g., a 1-in-50-year storm or a 40% stock market crash).
These tests are vital for understanding a model's stability. A well-built model should not produce wildly different results from a tiny change in an input. If it does, it may indicate instability or an implementation error.
[PDF] Model Validation Best Practices
This whitepaper provides a contemporary view on validation best practices, including a concise summary of testing techniques.
Please read the section titled "Output Review," from the beginning of the section. This part explains the purpose of stability checks, the different forms of stress testing, and introduces static vs. dynamic validation.
2. Back-Testing
Back-testing assesses a model's historical performance by comparing its predictions for a past period with what actually happened. For financial models, this requires a specific approach known as out-of-time testing.
Given your background in statistics, you're likely familiar with splitting data into training and testing sets for cross-validation. In many fields, this is done by random sampling ("out-of-sample" testing). However, for financial and economic data, which exhibits serial correlation (i.e., today's value is related to yesterday's), this is invalid. We cannot use information from the future to predict the past, even if it's just a few data points.
Therefore, we must use out-of-time testing:
- Develop and calibrate the model using data up to a certain point in time (e.g., through to the end of 2022). This is the "development" or "training" period.
- Use the finalized model to make predictions for a subsequent period (e.g., 2023). This is the "out-of-time" or "back-testing" period.
- Compare the model's predictions for 2023 with the actual outcomes that occurred.
This video, "Back Testing in Model Dev" by Dimitri Bianco, explains this concept brilliantly. It clarifies why out-of-time testing is non-negotiable in finance and introduces the critical concept of the "point of no return."
Please watch two key segments: Out of Sample vs. Out of Time: This explains why serial correlation makes random sampling invalid for financial models and establishes the necessity of a chronological data split. The Point of No Return: This segment discusses the crucial discipline of back-testing. Once you use the out-of-time data to test your final model, you cannot go back and tweak the model based on those results. Doing so introduces information from the future into your model development, a form of "cheating" known as information leakage.
The "point of no return" is a fundamental principle of good validation discipline. The out-of-time data set serves as a final, independent arbiter of the model's performance. It can be used to select the best model from a list of pre-developed candidates, but not to change any of them.
Independence and Challenger Models
A recurring theme in all the resources is independence. The validation should be performed by people who were not involved in developing the model. This ensures an unbiased and challenging review.
The most robust form of validation, often seen as the "gold standard," involves developing a challenger model. This is an independently built model, often simpler, used to benchmark the results of the primary model. If the two models produce wildly different results, it signals that a deeper investigation is required.
[PDF] Model Validation Best Practices
The "Model Validation Best Practices" whitepaper elaborates on this gold standard approach.
Please read the section "Calculation Review" starting from the first paragraph. It describes the use of first-principles or challenger models for comparative testing.
Conclusion
Model validation is not a one-off task but a continuous cycle of review and challenge that is fundamental to actuarial professionalism. It is a structured process designed to identify and manage model risk, ensuring that the models we rely on are fit for purpose. The process is both qualitative—assessing the conceptual underpinnings, documentation, and inputs—and quantitative, using powerful techniques like sensitivity analysis and back-testing to probe a model's behaviour and predictive power.
As we've seen, effective validation hinges on key principles: a structured approach covering all sources of risk, rigorous quantitative testing, and, critically, independence.
In our next lesson, we will build directly on the topic of Input Risk that we've touched on today. We will explore the requirements for data quality in detail, looking at the processes for ensuring data is accurate, complete, and appropriate for the modelling task at hand.
Key Takeaways & Cheatsheet
This cheatsheet summarizes the key areas of a validation process, aligned with the five categories of model risk. It can serve as a mental checklist when considering how to validate any given model.
| Risk Category | Key Question | Common Validation Checks |
|---|---|---|
| 1. Conceptual Risk | Is the model's theory and design appropriate for the intended use? | - Review model methodology documentation against industry standards and academic literature. - Assess if all material risks are captured. - Check for documented limitations. |
| 2. Input Risk | Is the data used by the model accurate, complete, and appropriate? | - Reconcile data to source systems. - Review data cleaning and adjustment processes. - Benchmark key parameters (e.g., assumptions) against industry data. |
| 3. Implementation Risk | Is the model built correctly in the software, free from material bugs? | - Use a challenger model for comparison. - Perform independent recalculations for sample policies. - Review code for clarity, efficiency, and adherence to standards. - Ensure proper version control and change management processes are in place. |
| 4. Output Risk | Are the model's results stable, reasonable, and robust? | - Back-testing: Compare model predictions against historical results using out-of-time data. - Sensitivity/Stress Testing: Analyze the impact of changing key assumptions. - Benchmarking: Compare outputs to simpler models or industry benchmarks. |
| 5. Reporting Risk | Are the results communicated to stakeholders clearly, without being misleading? | - Review management reports for clarity and completeness. - Ensure limitations and uncertainty are communicated alongside results. - Verify that the model is being used in decision-making as intended (the "Use Test"). |