Marketing vs. Evidence: AI Claims

The Two Types of Claims: Marketing vs. Evidence

When you encounter a new dental AI product, you'll be met with two kinds of statements: those designed to persuade you and those designed to prove something to you. Understanding the difference is your first line of defense against adopting ineffective or unreliable technology.

1. The Marketing Claim

Marketing claims are designed to be appealing and easy to understand. They often use exciting language, focus on high-level benefits, and present impressive-looking numbers without much context.

Let's look at a typical example.

This ad is a perfect illustration of marketing claims. It's visually appealing and makes the product sound highly effective. But as a critical evaluator, you should immediately ask:

• "95% AI Accuracy Rate": Accuracy at doing what? Detecting caries? Measuring bone loss? Numbering teeth?
• What does "accuracy" even mean here? As we discussed in Module 3, overall accuracy can be a misleading metric. Is this sensitivity, specificity, or something else?
• On what kind of data was this measured? Radiographs from one clinic or thousands? Were they from a diverse patient population?
• Compared to what? Was the AI's performance measured against a single dentist's opinion or a consensus of multiple expert radiologists?

Marketing claims are not necessarily false, but they are incomplete. They are the start of a conversation, not the end.

2. The Evidence-Based Claim

In contrast, an evidence-based claim is specific, verifiable, and provides the context needed for you to make a professional judgment. One of the strongest forms of evidence a company can offer is regulatory clearance.

The claim "FDA-cleared" is fundamentally different from "95% accuracy." It’s a statement of fact that can be verified and implies that the manufacturer has submitted performance data to a government body for review.

Why Regulatory Clearance Matters

For medical devices, including software, regulatory bodies like the U.S. Food and Drug Administration (FDA) act as crucial gatekeepers. Their purpose is to ensure that products sold to clinicians are safe and effective for their intended use.

To understand why this is so important for evaluating AI products, let's read the introduction to a recent review paper that cataloged all FDA-cleared AI solutions in dental imaging.

As the paper highlights, FDA clearance distinguishes clinically validated AI platforms from experimental technologies. To get this clearance, a manufacturer can't just make a marketing claim; they must submit detailed evidence demonstrating their product's performance.

Let's take a look at the kind of evidence they provide. The same paper includes tables summarizing the information submitted to the FDA.

• Table 1 lists the specific products and even the individual modules within a product that have received FDA clearance.
• Table 3 details the actual performance metrics from the validation studies.

The data in Table 3 is a prime example of an evidence-based claim. For instance, instead of a vague "high accuracy," Overjet's submission for its Caries Assist module shows:

• Standalone Performance: Sensitivity of 72.0% and Specificity of 98.1%.
• Reader Study Performance: When dentists used the AI, their sensitivity improved from 57.9% to 76.2%, while specificity only slightly decreased (from 99.3% to 98.4%).

This is a rich, nuanced, and verifiable claim. It tells you what the AI does, how well it performs on its own, and how it impacts a clinician's diagnostic performance. This is the kind of data you need to make an informed decision.

From Marketing to Evidence: Your Evaluation Framework

So, when you encounter a new product, how do you move past the marketing and get to the evidence? You can use a simple fact-checking process, applying the critical appraisal skills you've already learned.

Let's begin by reviewing a simple framework for appraising clinical papers, which we can adapt for our purpose.

Here is a practical framework for evaluating a company's claims, inspired by the video and our previous lessons:

Step 1: Identify a Specific, Verifiable Claim
Look past slogans and find a concrete statement. For example, in a popular TEDx talk, the CEO of VideaHealth makes a very specific claim.

The claim is: "we were able to reduce the missed cavities by 43%." This is a number that must have come from a study.

Step 2: Ask for the Evidence
Your next question should always be: "Where is the data that supports this claim?" The evidence could be:

• An FDA 510(k) summary document.
• A peer-reviewed publication in a reputable journal.
• A company "white paper" (be cautious with these, as they are not peer-reviewed).

Step 3: Scrutinize the Evidence
This is where you put your research appraisal skills to work.

• Is the product FDA-cleared for that specific claim? Not all products on the market are. The review paper we studied earlier includes a table of commercially available platforms without verified FDA clearance.

• Does the evidence stand up to scrutiny? If the company provides a study, ask the questions from Module 8:
  • Was there an independent test set?
  • Was the study conducted on data from multiple centers?
  • Are there signs of bias in the data?
  • How generalizable are the findings to your patient population?

The FDI World Dental Federation's white paper on AI emphasizes that clinicians have a responsibility to critically appraise these tools.

As the FDI paper states, "the final responsibility for any decisions emanating from using a diagnostic support system remains with the user, i.e. the oral healthcare provider." This underscores why these evaluation skills are not just academic, but essential for professional practice.

Conclusion

You are now equipped with the foundational skill for evaluating any commercial AI product: the ability to look past the marketing and demand evidence.

Key Takeaways:

• Marketing claims are often vague, persuasive, and lack context (e.g., "95% accuracy").
• Evidence-based claims are specific, verifiable, and tied to data from studies or regulatory submissions (e.g., "In a multi-reader study, sensitivity for caries detection improved from 65% to 80%").
• FDA clearance is a critical indicator of an evidence-based product, as it requires manufacturers to submit performance data for review.
• Many AI products are marketed without having FDA clearance.
• Your job as a clinician is to ask for the evidence and use your critical appraisal skills to judge its quality, paying close attention to the study design, data, and generalizability.

Preview of the Next Lesson

In this lesson, we learned what to look for. In the next lesson, we will learn how to find it. We will focus on the next step in our evaluation framework: "Cross-reference a manufacturer's claims with their published validation studies or regulatory filings (e.g., FDA, CE)." We will take a hands-on look at how to search the FDA database and dissect the summary of a real AI product's submission.