Hello! Welcome to your next lesson in the "Foundations of Clinical Research and Biostatistics" module.
In our last lesson, we learned how to select the right statistical test, like a t-test or chi-square test, to see if an observed difference between two groups is "statistically significant." We saw this in action with the Musta Churna study, where the reduction in cholesterol was highly significant (p < 0.001). This p-value told us the result was very unlikely to be due to chance.
However, statistical significance is only half of the story. A result can be statistically significant but practically meaningless. This brings us to today's learning outcome: to explain the importance of effect size in conjunction with statistical significance. We will explore how to answer the crucial follow-up question: "Okay, the effect is real, but is it big enough to matter?" For your goal of critically evaluating research on Ayurvedic treatments, this distinction between a statistical effect and a clinically meaningful one is paramount.
Part 1: The Difference Between "Different" and "How Different"
A small p-value gives us confidence that a difference between two groups exists. What it doesn't tell us is the magnitude of that difference. A small p-value can correspond to a tiny, trivial effect or a huge, life-changing one.
Let's begin with a short video that makes this exact point.
p-values: What they are and how to interpret them
The video 'p-values: What they are and how to interpret them' by StatQuest with Josh Starmer concludes with a crucial clarification about what p-values do and do not tell us.
Watch the segment from 09:41 to 11:02. Pay close attention to the conclusion that a small p-value does not imply a large effect size.
This video highlights a key limitation of relying solely on p-values. The statistical significance is influenced by the number of participants in a study. With a large enough sample size, even a minuscule, practically irrelevant difference can become statistically significant.
This is the distinction between statistical significance (is the effect likely real?) and practical significance (is the effect large enough to be useful in the real world?).
The next video explains this distinction using some great examples.
Degrees of Freedom and Effect Sizes: Crash Course Statistics #28
The video 'Degrees of Freedom and Effect Sizes' from CrashCourse Statistics introduces the concept of practical significance and explains why it's a necessary partner to statistical significance.
Watch the section from 06:52 to 09:16. The video uses examples of bee populations and a hair growth vitamin to illustrate that a statistically significant effect may not be practically important.
Part 2: Defining and Understanding Effect Size
So, if p-values tell us about statistical significance, how do we measure practical significance? The answer is effect size.
Effect size is a quantitative measure of the magnitude of a phenomenon. It tells us how big the difference between groups is or how strong the relationship between variables is. Unlike a p-value, it is independent of sample size.
Let's dive into some reading to solidify this concept.
What is Effect Size and Why Does It Matter? (Examples)
The article 'What is Effect Size and Why Does It Matter?' from Scribbr provides a very clear and concise introduction to this topic.
Read the sections 'Why does effect size matter?' and the FAQ at the end, 'What’s the difference between statistical and practical significance?'. This will reinforce the core distinction we've been discussing.
To add a bit more detail, there are two main ways to think about effect size:
-
Absolute Effect Size: This is the simple, raw difference between the average outcomes of two groups.
- Example: In a study on hypertension, if the treatment group's blood pressure dropped by an average of 15 mmHg and the placebo group's dropped by 5 mmHg, the absolute effect size is a 10 mmHg difference. This is easy to understand when the units (like mmHg) have intrinsic meaning.
-
Standardized Effect Size: This transforms the effect into a common scale, which allows us to compare results across different studies that might have used different measurement tools. A very common standardized measure is Cohen's d.
This next reading explains these types and delves deeper into why the p-value isn't enough.
Using Effect Size—or Why the P Value Is Not Enough - PMC
The academic paper 'Using Effect Size—or Why the P Value Is Not Enough' provides a more formal explanation, including a powerful real-world example.
Please read the sections 'What Is Effect Size?', 'Why Report Effect Sizes?', and 'Why Isn't the P Value Enough?'. The aspirin study example in the last section is a classic illustration of a highly significant p-value paired with a very small effect size.
Part 3: Interpreting Effect Size: Small, Medium, or Large?
Once we have a standardized effect size like Cohen's d, we need to interpret its magnitude. Jacob Cohen, a statistician, provided widely used "rules of thumb" for this:
| Cohen's d | Interpretation |
|---|---|
| ~ 0.2 | Small effect |
| ~ 0.5 | Medium effect |
| ~ 0.8+ | Large effect |
- A small effect (d = 0.2) is noticeable but not dramatic. The distributions of the two groups overlap significantly.
- A medium effect (d = 0.5) is large enough to be "visible to the naked eye." The average person in the treatment group would score higher than 69% of the people in the control group.
- A large effect (d = 0.8) represents a substantial difference between the groups, with less overlap in their distributions. The average person in the treatment group would score higher than 79% of the control group.
It's important to remember these are general guidelines. The "importance" of an effect size also depends on the context. In a life-threatening disease, a small effect could still save many lives. For an Ayurvedic supplement for mild fatigue, one might hope for at least a medium effect to justify its use.
Test your understanding!
Imagine you are reading two different studies on a new herbal preparation for improving memory scores.
- Study A: Finds a statistically significant improvement in memory scores (p = 0.02) with a Cohen's d of 0.15.
- Study B: Finds an improvement that is not statistically significant (p = 0.10) with a Cohen's d of 0.70.
Which study's findings are more promising for future research, and why?
Show answer
Study B is more promising. Although it failed to find a statistically significant result (perhaps due to a small sample size, leading to a Type II error), the effect size is large (0.70). This indicates the herbal preparation may have a substantial, practically meaningful effect on memory. The intervention in Study A, while statistically significant, has a very small effect size (0.15), suggesting the improvement it causes is trivial and may not be practically relevant. Study B's findings strongly warrant a follow-up study with a larger sample size to gain more statistical power.
Part 4: A Tale of Two Significances
Let's watch a final video clip that perfectly illustrates how statistical significance without a meaningful effect size can be misleading.
Degrees of Freedom and Effect Sizes: Crash Course Statistics #28
Let's return to the CrashCourse Statistics video to see a fantastic, extended example of how a massive sample size can create a statistically significant result for a practically useless intervention.
Watch the segment from 09:04 to 11:57. The 'WOWZERBRAIN!' supplement example shows how a tiny effect becomes 'significant' simply by increasing the number of participants, while the effect size remains small and unimportant.
This example drives home the main point: p-values are not everything. You can achieve statistical significance by simply collecting more and more data, but that doesn't make the effect itself any larger or more important.
Let's apply this to an Ayurvedic context.
- Scenario 1: A study with 50,000 people finds that taking a pinch of a specific herb daily reduces the risk of catching a common cold by 0.5% over a year. Because the sample size is huge, the p-value is tiny (p < 0.0001). The result is statistically significant, but the effect size is so small it's clinically meaningless.
- Scenario 2: A pilot study of 30 people finds that a two-week regimen of Guduchi reduces fever duration by an average of 2 days. Because the sample is small, the p-value is 0.09 (not statistically significant). However, reducing fever by 2 days is a large effect size and is clinically very meaningful. This promising result justifies a larger, more powerful study.
Conclusion
Today we've learned that to properly interpret research, we must act as detectives looking for two key pieces of evidence: the p-value and the effect size. One without the other provides an incomplete and potentially misleading picture.
Key Takeaways:
- Statistical Significance (p-value): Tells you if an effect is likely due to the intervention rather than random chance. It is heavily influenced by sample size.
- Practical Significance (Effect Size): Tells you the magnitude or importance of the effect. It is independent of sample size.
- A research finding is most compelling when it is both statistically significant (e.g., p < 0.05) and has a practically meaningful effect size (e.g., medium to large).
- Always be skeptical of claims of "significance" that do not also report the size of the effect.
Preview of the Next Lesson:
Now that we can critically evaluate the results of a study by considering both its statistical and practical significance, we will turn our attention to the foundation upon which all ethical research is built. In our next lesson, we will describe the core ethical principles governing human research, including informed consent, beneficence, and justice. Understanding these principles is a non-negotiable prerequisite for anyone aspiring to conduct or even interpret clinical studies.