Communicating Model Results: Limitations and Uncertainty

The Foundation: Why We Must Communicate Uncertainty

The professional obligation to communicate uncertainty is not merely good practice; it is a core tenet of actuarial work. The aim is to ensure that anyone using actuarial information can rely on it. This is formally captured in what is known as the "Reliability Objective".

The key phrase here is the need to include "the communication of any uncertainty inherent in the information." This means that expressing uncertainty is not an optional extra or an admission of failure; it is a mandatory and integral part of the output itself. Failure to do so could lead to a user placing false confidence in a model's output, potentially resulting in poor business decisions.

From Statistical Concepts to Business Language

To communicate uncertainty effectively, we first need a clear and intuitive way to think about it. While you are well-versed in the mathematics of probability, the challenge is to convey these ideas without resorting to technical jargon.

The dartboard analogy powerfully illustrates several key concepts you must be able to explain:

• Uncertainty as a Range: A model doesn't produce "the" answer; it produces a range of plausible answers. The wider the spread of darts on the board, the higher the uncertainty.
• The Confidence-Precision Trade-off: This is a crucial concept. We can be highly confident in a very imprecise statement (e.g., "The hurricane will land somewhere in North America"). Conversely, a very precise prediction ("The hurricane will make landfall at this exact coordinate") carries very low confidence. Your job is often to help stakeholders find the right balance for their decision-making needs.
• Confidence Level as a Choice: The 95% confidence level is a convention, not a law of nature. The appropriate level of confidence depends on the decision being made. A decision with severe consequences for being wrong requires a higher level of confidence, which, as we've seen, will necessitate a wider, less precise range.

Tailoring Communication to the Audience

Effective communication is never one-size-fits-all. The content, style, and level of detail must be tailored to the audience. A report for a regulator will differ significantly from a presentation to the Board or a discussion with an underwriter.

We can broadly categorize stakeholders into internal and external groups, each with different information needs.

A working party from the Institute and Faculty of Actuaries has provided a useful guide on what to communicate and to whom.

Based on this, here's a summary of the key elements to communicate:

Navigating Stakeholder Personalities

Beyond their formal roles, stakeholders are individuals with different cognitive styles and attitudes toward models. A powerful framework for understanding these differences uses two axes: concern for model uncertainty and the perceived legitimacy of modeling.

Let's break down how to approach each type:

1. Confident Model Users (High Legitimacy, Low Concern):

   • Mindset: "Known Knowns." They trust the model and seek optimal, precise answers.
   • Risk: They might ignore limitations and treat the model output as truth.
   • Your Strategy: Don't just present the final number. Emphasize the key assumptions that the "optimal" answer depends on. Use sensitivity analysis to show how the answer changes if those assumptions prove wrong. Frame it as "The model recommends X, and this holds true as long as Y and Z are in these ranges."

2. Conscientious Modellers (High Legitimacy, High Concern):

   • Mindset: "Known Unknowns." These are often your fellow technical experts. They are aware of model limitations and want to understand them fully.
   • Your Strategy: You can be more technical with this group. Discuss the modeling choices made, the trade-offs involved (e.g., why you chose a simpler, proportionate approach), and the results of validation tests. The main challenge is to avoid "analysis paralysis" and ensure the discussion leads to a decision.

3. Intuitive Decision Makers (Low Legitimacy, Low Concern):

   • Mindset: "Unknown Knowns." They trust their gut and experience ("market realities") over a "black box" model.
   • Your Strategy: Do not lead with the model. Lead with their intuition. Frame the conversation around the business problem. Use the model to support, challenge, or provide a new perspective on their intuition. For example: "You've mentioned that you feel we are under-pricing this risk. The model suggests a similar conclusion, and it highlights that the main driver is the recent spike in claims from this specific region."

4. Uncertainty Avoiders (Low Legitimacy, High Concern):

   • Mindset: "Unknown Unknowns." They are wary of models and highly concerned about what could go wrong. They seek robustness and safety.
   • Your Strategy: Focus on risk mitigation. Use scenario analysis to present a range of outcomes, including plausible pessimistic scenarios. This addresses their concern for the unknown by making it more concrete. Explain how the model helps to quantify the potential impact of these risks, enabling the business to prepare for them.

Practical Application: How to Say It

Having the right framework is one thing; finding the right words is another. Let's look at some concrete examples of how to put these principles into practice.

A common situation is having to present results based on incomplete data or analysis, where you cannot be 100% certain.

This simple example demonstrates a key skill: being clear about what the data suggests versus what it proves, and providing enough evidence to justify a business decision despite the residual uncertainty.

The professional guidance for actuaries is full of scenarios that reinforce these communication duties under various real-world pressures like tight deadlines, incomplete data, or complex group dynamics.

These scenarios reinforce a consistent theme: your duty is to ensure the user of your work understands its context, limitations, and the uncertainty of the results, enabling them to make a well-informed decision.

Conclusion

This lesson concludes our module on modeling and data. We have journeyed from the purpose of models, through the practicalities of data quality and the principle of proportionality, to this final, crucial step of communication. The central message is that a model's output is not the end of the story; it is the beginning of a conversation with a stakeholder.

Effective communication of model results is a blend of technical honesty and audience awareness. It requires you to translate statistical concepts into business-relevant insights, to be transparent about limitations and assumptions, and to tailor your message to the person receiving it. By mastering this skill, you transform your technical analysis from a complex calculation into a powerful tool for strategic decision-making.

Looking ahead, our next module will delve into Assumption Setting and Analysis. This is a natural progression, as the assumptions you choose—about mortality, expenses, investment returns, and more—are the primary drivers of the model uncertainty that you must so carefully communicate.

Key Takeaways & Cheatsheet

Here is a framework for structuring your communication about model results.