Recommended resources
The freely available resources this lesson will direct you to.
Hello! Welcome to the third lesson in our course on AI governance.
In our last session, we differentiated between the governance challenges of predictive and generative AI, noting that while one amplifies existing data risks, the other introduces entirely new categories of concern. We briefly touched upon risks like bias and performance drift. Today, we will build on that foundation to formally define these concepts.
This lesson focuses on the learning outcome: Define key AI-specific risks such as model bias, opacity, and data drift within the AI lifecycle. Your experience in IT governance has given you a strong framework for thinking about risk across a system's lifecycle. Here, we'll apply that same structured thinking to the unique risks inherent in AI systems, equipping you with the precise vocabulary needed to identify, assess, and discuss these issues with both technical and business stakeholders.
1. An Overview of Core AI Risks
Before we dive deep into each risk, let's start with a high-level overview. Many of the most common AI failures you hear about—from biased outcomes to nonsensical chatbot responses—can be traced back to a handful of fundamental risks.
The following video provides a concise introduction to several of these, directly touching on the three we will focus on today.
The Importance of AI Governance
This clip from the IBM Technology channel, "The Importance of AI Governance," gives a clear, business-oriented primer on why these risks matter.
Please watch the following segments: Bias: Notice how the speaker links model bias directly to the human-generated data used for training. Transparency: Pay attention to the explanation of "black box" models and how their lack of transparency erodes trust. Model deterioration: This section introduces the concept that models are not static and can degrade over time, which we will explore as "data drift."
This video sets the stage nicely. Now, let's dissect each of these risks in more detail, placing them within the context of the AI development and deployment process.
2. Model Bias: Where and How It Enters the Lifecycle
In the context of AI, bias refers to a systematic deviation in a model's outputs that leads to unfair, inequitable, or discriminatory results for certain individuals or groups. It's one of the most significant reputational and regulatory risks in AI governance.
A common misconception is that bias is purely an algorithmic problem. In reality, it can be introduced at nearly every stage of the AI lifecycle.
Five AI risks IT professionals should spot before deployment - BCS
This section of the BCS article "Five AI risks IT professionals should spot before deployment" provides an excellent professional summary of model bias.
Read the section titled "One: bias in data and algorithms." Focus on how it frames bias as a technical and reputational risk and points to training data as a primary source.
The article correctly identifies that the roots of bias are often found in data. The diagram below provides a powerful illustration of this, mapping specific types of bias to the stages of the AI model lifecycle. This is a crucial mental model for a governance professional.
Let's walk through a few examples from the "AI Model Life Cycle" diagram:
- Conception: If the initial problem is framed in a biased way (e.g., "predicting criminality" based on neighborhood data), the entire project may have an inherent Systemic Bias.
- Data Collection: If a dataset for a facial recognition model predominantly features light-skinned individuals, it suffers from Representation Bias. The model will likely perform poorly for underrepresented groups.
- Pre-processing: If a data scientist chooses to handle missing income data by filling it with the average income of a particular zip code, they might introduce bias against residents of lower-income areas.
- Post-deployment Surveillance: Even if a model is fair at launch, if it isn't monitored, it can start producing biased outcomes as user demographics or behaviors change over time.
Your role in governance is to ensure that checks for these types of biases are embedded as formal controls at each stage gate of the lifecycle.
3. Opacity: The "Black Box" Problem
Opacity refers to the difficulty, or sometimes impossibility, of understanding the internal workings of an AI model. This is also known as the "black box" problem. While some simple models are interpretable (e.g., decision trees), the most powerful models, like deep neural networks, often operate in ways that are not fully understood even by their creators.
This poses a direct challenge to governance principles you're familiar with, such as auditability and accountability.
Five AI risks IT professionals should spot before deployment - BCS
Let's return to the BCS article to define this risk.
Please read the section "Three: lack of explainability and transparency." It highlights why opacity is particularly risky in regulated sectors and emphasizes designing for interpretability from the start.
The governance implications of opacity are profound. If you can't explain why a model denied someone a loan, you can't demonstrate compliance with fair lending laws. If you can't trace a model's decision pathway, you can't effectively investigate or remediate an AI-related incident.
This risk is further complicated by a phenomenon described as Black-Box Evolution.
[PDF] AI IDENTITY DRIFT - Phase 5
This whitepaper from Phase 5 offers a sophisticated look at the causes of AI risk.
Find the section titled "HOW AI IDENTITY DRIFT BEGINS: THE STRUCTURAL CAUSES." Read the subsection "4. Black-Box Evolution – When AI Outpaces Human Oversight." This provides a deeper insight into how opaque models can change their internal logic without a visible trail.
As the paper explains, the model's internal logic can shift in ways that are not legible. This reinforces the need for governance to focus on monitoring the model's outputs and behavior rather than relying solely on auditing its internal code or logic, which may not be possible.
4. Data Drift: When a Good Model Goes Bad
Finally, we come to data drift. This is the single biggest reason for model performance degradation in production. Data drift occurs when the statistical properties of the data the model sees in the real world change from the data it was trained on.
An AI model is a snapshot in time. It learns patterns from its training data. When the world moves on, but the model doesn't, its predictions become less accurate and less reliable. This is not an "if" but a "when."
ML Drift: Identifying Issues Before You Have a Problem
This video from Fiddler AI, titled "ML Drift," is dedicated entirely to this topic and provides an excellent, practical breakdown.
Watch the following segments for a detailed explanation: What is drift?: This introduces the core concept and highlights its significant business impact. Types of drift: This is the most critical part. Pay close attention to the distinction between concept drift, label drift, and feature drift, using the loan application example.
The video provides the level of detail a governance professional needs. Let's summarize the key types of drift it introduced:
| Type of Drift | Description | Loan Application Example (from the video) | Governance Implication |
|---|---|---|---|
| Concept Drift | The relationship between inputs and outputs changes. The meaning of what you're predicting evolves. | Due to a recession, an income level once considered 'creditworthy' is now considered 'high-risk'. The concept of "creditworthiness" has changed. | Requires monitoring real-world outcomes and may trigger a full model retrain with new business logic. |
| Feature Drift | The distribution of the input data changes. You're seeing a different type of data than what the model was trained on. | The average income level of applicants suddenly increases because of a new marketing campaign in an affluent area. | Requires monitoring input data distributions. It's an early warning that model performance might degrade soon. |
| Label Drift | The distribution of the output data (the "ground truth") changes. | A much higher percentage of applicants are now creditworthy than in the training data. | Requires monitoring the distribution of actual outcomes. It can signal that the business environment has fundamentally shifted. |
From a governance standpoint, the critical takeaway is that deploying an AI model is not the end of the process. A robust governance framework must include continuous monitoring for these different types of drift to detect performance degradation before it leads to financial loss or harmful outcomes.
Conclusion
In this lesson, we have moved beyond a general awareness of AI risk to a more formal, structured understanding of three critical issues.
Here are your key takeaways:
- Model Bias is a systemic risk that can originate from biased data or flawed human decisions at any stage of the AI lifecycle, from initial conception to post-deployment monitoring.
- Opacity (the "black box" problem) makes it difficult to audit or explain a model's decisions, posing a fundamental challenge to accountability and regulatory compliance.
- Data Drift is the inevitable degradation of a model's performance as the real-world data it encounters diverges from its training data. It comes in specific, monitorable forms like concept, feature, and label drift.
You now have a clear definition of what you need to govern. In our next lesson, we will focus on how to govern it. We will begin mapping the familiar controls and processes from your IT governance background to their new AI governance counterparts, addressing the specific risks we've defined today.