Evidence-Based Claim Evaluation

1. Defining "Sufficient and Appropriate" Evidence

At its core, evaluating evidence means asking two fundamental questions: "Is it the right kind of evidence?" (appropriateness) and "Is there enough of it?" (sufficiency).

Let's begin with a concise guide that frames these questions.

This guide gives us a high-level checklist. Now, let's deepen our understanding of sufficiency by looking at a more formal definition.

The resource highlights three crucial points about sufficiency:

1. The burden of proof rests on the claimant. It is not our job to disprove a claim; it is the claimant's job to provide enough evidence to prove it.
2. Extraordinary claims demand extraordinary evidence. A mundane claim requires less proof than a revolutionary one.
3. Evidence based on authority or testimony is inadequate for most claims. While expert opinion has its place, it is not a substitute for empirical data.

Let's apply this:

• Bra Design Claim: A designer claims, "My new bra design, featuring asymmetrical cups, is more comfortable for 90% of women than any symmetrical design." This is an extraordinary claim because it challenges fundamental conventions of bra construction.
  • Inadequate Evidence: Testimonials from a few satisfied customers.
  • Sufficient Evidence Required: This would demand large-scale, controlled studies comparing this design against multiple standard designs across a wide range of body types, with comfort measured using validated, objective scales over extended wear periods.

2. The Hierarchy of Evidence: Assessing Appropriateness

The "appropriateness" of evidence is about its quality and strength, which is largely determined by the study design used to collect it. Not all research designs are created equal when it comes to proving a claim, especially a causal one. This is captured in the "Hierarchy of Evidence."

This hierarchy provides a framework for weighing evidence. A randomized controlled trial (RCT) provides much stronger evidence for a causal link than a collection of case studies or an expert's opinion.

Let's illustrate this with a claim from your interest in relationship dynamics.

Claim: "In single-female-multiple-male polyamorous relationships, a structured weekly 'relationship check-in' meeting prevents relationship dissolution."

We can evaluate the evidence for this claim based on where it falls in the hierarchy:

• Level VI (Opinion): A well-known author on polyamory writes a blog post stating that check-ins are essential. This is an idea, a hypothesis, but it is not strong evidence.
• Level IV (Case-Control Study): Researchers identify 50 dissolved triads ("cases") and 50 intact triads ("controls") and ask them about their past meeting habits. This is weak evidence, highly susceptible to recall bias and confounding factors (perhaps the triads that had meetings were also more communicative in general).
• Level III (Cohort Study): A study follows 100 new triads over five years. It records which ones naturally adopt a weekly check-in and compares their dissolution rates to those who don't. This is stronger, as it establishes a temporal sequence (the check-ins precede the outcome). However, it's still prone to self-selection bias.
• Level II (Randomized Controlled Trial): In an ideal (though practically difficult) study, researchers would recruit 100 new triads and randomly assign 50 to a "check-in" protocol and 50 to a "no structured meeting" control group. They would then follow both groups for five years. This design, by randomizing, minimizes the influence of confounding variables and provides the strongest evidence for a causal link from a single study.
• Level I (Systematic Review): An expert group gathers all the high-quality studies (ideally, multiple RCTs) on this topic, assesses their quality, and synthesizes the results. This provides the most reliable conclusion.

When you evaluate a claim, identifying the type of study used to generate the evidence immediately tells you a great deal about its potential strength and appropriateness.

3. Quantifying the Evidence: Measuring Sufficiency

Once we've established that the evidence is of an appropriate type, we must assess if it's sufficient in quantity and magnitude. This is where your background in statistics becomes paramount. We need to look beyond the headlines and examine the numbers.

Statistical Significance vs. Practical Importance

Your graduate students will know that a "statistically significant" result (e.g., $p<0.05$) simply suggests that the finding is unlikely to be due to random chance. It does not, however, say anything about the size or importance of the effect.

Bra Design Example:
Imagine a study testing a new bra strap material.

• Claim: "Our new 'Flex-Strap' material significantly reduces shoulder pressure."
• Evidence: An RCT with 5,000 participants finds that the Flex-Strap reduces pressure by an average of 0.5 grams/cm². Because of the huge sample size, the result is statistically significant ($p=0.01$). The 95% confidence interval for the pressure reduction is [0.1, 0.9] grams/cm².
• Evaluation: The evidence is statistically sufficient to show a real, non-chance effect. However, is a reduction of 0.5 grams/cm² a meaningful improvement in comfort? Likely not. The effect size is tiny, so the evidence is not sufficient to support a claim of a major comfort breakthrough.

Quantifying Effect Size

To properly judge sufficiency, we need metrics that describe the magnitude of the effect.

Let's apply these powerful concepts to a polyamory-related claim.

Claim: "A new 'Compersion-Focused Therapy' (CFT) program helps polyamorous individuals overcome jealousy."

• Evidence: A cohort study follows individuals starting polyamorous relationships. 20% of those who do not attend CFT report significant jealousy issues after one year, compared to only 10% of those who do attend CFT.

How do we evaluate the sufficiency of this evidence?

1. Relative Risk (RR): The risk of jealousy for the CFT group is 0.10, and for the control group is 0.20. The Relative Risk is $0.10/0.20=0.5$. We can say "CFT cuts the risk of jealousy in half." This sounds very impressive.

2. Absolute Risk Reduction (ARR): The absolute difference in risk is $0.20-0.10=0.10$, or 10 percentage points. This is a more sober assessment of the actual impact.

3. Number Needed to Treat (NNT): This is perhaps the most intuitive measure of practical sufficiency. It is the inverse of the ARR.

   $$\text{NNT}=\frac{1}{\text{ARR}}=\frac{1}{0.10}=10$$

   This means you would need to treat 10 people with the Compersion-Focused Therapy to prevent one additional person from experiencing significant jealousy.

The NNT provides a clear, tangible metric for judging sufficiency. Is it worth the time and expense of putting 10 people through therapy to help one of them? That's a value judgment, but the NNT grounds the decision in a concrete number, stripping away the marketing gloss of "cutting the risk in half."

Conclusion

In this lesson, we have armed ourselves with a robust toolkit for evaluating the evidence behind a claim. We've learned that it's not enough for evidence to be objective; it must also be appropriate for the claim and sufficient to be convincing.

Key Takeaways:

• Evidence must be evaluated for both appropriateness (the quality and type of data) and sufficiency (the quantity and magnitude of the data).
• The Hierarchy of Evidence provides a framework for judging appropriateness, ranking study designs like RCTs and systematic reviews as stronger than observational studies or expert opinion.
• Sufficiency is not just about statistical significance ($p$-values). We must also assess the practical importance by looking at the effect size.
• Metrics like Absolute Risk Reduction (ARR) and Number Needed to Treat (NNT) are powerful tools for quantifying the real-world impact of an intervention and judging the practical sufficiency of the evidence.
• Remember: Extraordinary claims demand extraordinary evidence.

Preview of the Next Lesson:
We have now scrutinized a claim's structure (Logic), the purity of its data (Objectivity), and the strength of its proof (Evidence). But what if we've missed something? What if another factor explains the results? In our next lesson, we will tackle the 'A' in FLOATER: Alternative Explanations. We will explore the critical role of confounding variables and the importance of considering other interpretations before accepting a claim.