Hello! Welcome to the third lesson in our module on AI performance metrics.
In our last two lessons, we defined the four core metrics for a diagnostic test: sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). You learned that sensitivity and specificity describe the test's inherent ability to detect disease or its absence, while PPV and NPV give us the practical probability of disease given a specific test result. Crucially, you saw how PPV and NPV are influenced by the prevalence of the condition in the population being tested.
This brings us to a metric you'll see everywhere: overall accuracy. It sounds straightforward—what percentage of the time was the AI right? However, today's learning outcome is to explain why overall accuracy can sometimes be a misleading metric. This is one of the most important concepts for critically evaluating any AI tool, especially in a clinical field like dentistry where the stakes are high.
The Allure and the Trap of Accuracy
At first glance, accuracy seems like the perfect measure. It's simple to calculate:
It answers the seemingly simple question, "How often is the model right?" While this is an intuitive starting point, it hides some dangerous pitfalls, especially in medicine.
Let's start with a very clear illustration of why a high accuracy score can be so deceptive.
Why 99% Accuracy Isn't Always Good Enough in Healthcare AI
This article, 'Why 99% Accuracy Isn't Always Good Enough in Healthcare AI,' uses a simple, powerful example to show how accuracy can fail us.
Read the introduction and the sections 'Why Accuracy Isn’t Enough (Especially in Healthcare)' and 'Why This is Dangerous'. Focus on the 'rare disease' example and the explanation of imbalanced data.
The Problem of Imbalanced Data in Dentistry
The article you just read introduces the core issue: imbalanced data. This is a situation where the classes you are trying to predict are not equally represented. In medicine and dentistry, this is the norm, not the exception.
Think about screening for a condition like oral cancer in your patient population. The vast majority of your patients will be healthy (the "negative" class), while a very small number may have the disease (the "positive" class).
- Healthy patients: 99.9%
- Patients with oral cancer: 0.1%
If an AI model simply learns to predict "healthy" for every single patient, it will be correct 99.9% of the time. Its accuracy score would be near-perfect, but the model would be clinically useless because its one job—to find the rare but critical cases of cancer—would fail every single time.
This problem of "unbalanced classes" is a well-known challenge in medical AI research.
Scope and challenges of machine learning-based diagnosis ... - PMC
A research paper on the challenges of dental AI discusses this exact problem, confirming its relevance in our field.
Please read the section titled '4.5. Unbalanced classes'. Note how it describes the problem in the context of medical diagnosis and specifically mentions oral cancer, explaining why algorithms that just maximize accuracy can produce misleading results.
As the paper states, a model that simply favors the majority class (e.g., "no cancer") might show a high accuracy value but give a "misleading understanding of the actual performance." It fails because accuracy treats a false negative (missing a real cancer case) as being no more important than a false positive (flagging a healthy patient for a follow-up). Clinically, we know these two errors have vastly different consequences.
This image provides a good visual summary of an imbalanced dataset.
The Medical Test Paradox Revisited
This issue connects directly to what we learned about prevalence and predictive values in the last lesson. A high overall accuracy doesn't guarantee that a positive test result is meaningful, especially when the condition is rare (low prevalence).
Let's watch a classic explanation of this paradox. It will link the concepts of sensitivity, specificity, prevalence, and accuracy together.
The medical test paradox, and redesigning Bayes' rule
This famous video from the channel 3Blue1Brown explains the 'medical test paradox'. It shows how a test can be very 'accurate' in terms of its technical specifications, yet still have a low probability of being correct when it returns a positive result.
Watch from 01:14 to 03:54. The video presents a concrete example of a breast cancer screening for 1,000 women. Pay close attention to how a test with high sensitivity (90%) and specificity (~91%) results in only a 1-in-11 chance of having cancer after a positive test. This is the paradox in action.
Let's break down the numbers from that video to calculate the overall accuracy:
- Total Population: 1,000 women
- True Positives (cancer, tested positive): 9
- True Negatives (no cancer, tested negative): 901
- False Positives (no cancer, tested positive): 89
- False Negatives (cancer, tested negative): 1
The overall accuracy is:
An accuracy of 91% sounds pretty good! But as the video powerfully demonstrates, if you get a positive result, your chance of actually having the disease (the PPV) is only 9 / (9 + 89) = 9/98, which is about 9.2% or 1 in 11.
This is the core reason why accuracy is misleading: a single high number (91%) obscures the much more important and sobering reality of the low predictive value (9.2%) for a positive result in a low-prevalence screening scenario.
Conclusion: Context is Everything
Relying on a single accuracy score is like summarizing a patient's entire health with a single number. It's an oversimplification that misses the nuances crucial for good clinical judgment.
Key Takeaways:
- Accuracy can be highly misleading for imbalanced datasets, where one class (e.g., healthy) is far more common than another (e.g., diseased). This is a frequent scenario in dental and medical screening.
- A model can achieve near-perfect accuracy by simply always predicting the majority class, making it clinically useless for detecting rare conditions.
- Accuracy gives equal weight to all types of errors, but in a clinical setting, a false negative (missing a disease) is often far more dangerous than a false positive (triggering an unnecessary follow-up).
- A high overall accuracy does not guarantee a high Positive Predictive Value (PPV), especially when disease prevalence is low.
This is why we need the other metrics you've learned. Sensitivity, specificity, PPV, and NPV force us to look at the model's performance from different angles, giving us a much richer and more clinically relevant picture.
In our next lesson, we will explore a powerful tool that helps us visualize the trade-off between a model's benefits (catching true positives) and its costs (producing false positives). We will learn to interpret a Receiver Operating Characteristic (ROC) curve and the meaning of the Area Under the Curve (AUC), which provides a single, more robust metric than accuracy for summarizing a model's performance across all possible thresholds.