Interpreting P-values and Confidence Intervals

1. The p-Value: Quantifying Surprise

When we conduct a study, we start with a default assumption called the null hypothesis ($H_0$). This hypothesis states that there is no effect or no difference between the groups being compared. For example, in a trial comparing a new Ayurvedic formulation to a placebo for improving sleep quality, the null hypothesis would be that the formulation has no effect on sleep quality.

The p-value is a probability that helps us decide whether to stick with this null hypothesis or reject it in favor of an alternative hypothesis ($H_1$), which states that there is an effect.

To get a clear and intuitive understanding of what a p-value is and how it's used, let's start with a short video.

Key Concepts from the Video

• Definition: The p-value is a number between 0 and 1. It measures how compatible your data are with the null hypothesis. A small p-value indicates that your observed data are surprising, or unlikely, if the null hypothesis were true.
• The Significance Threshold ($\alpha$): In most medical research, a threshold (called alpha, $\alpha$) is set at 0.05.
  • If p ≤ 0.05, we reject the null hypothesis. We conclude that the result is statistically significant, meaning the observed difference is unlikely to be due to random chance alone.
  • If p > 0.05, we fail to reject the null hypothesis. The result is not statistically significant. We don't have enough evidence to say the observed difference wasn't just due to chance.
• Important Nuance: Failing to reject the null hypothesis does not prove that it's true. It simply means we lack sufficient evidence to say it's false. This is like a court verdict of "not proven guilty" rather than "innocent."

2. The Critical Limitation of the p-Value

The p-value tells you whether an effect is likely to be real (i.e., not due to chance), but it tells you nothing about the size or the clinical importance of that effect. This is a point of paramount importance for clinicians.

Let's continue with the StatQuest video, which makes this exact point.

A study with a very large sample size can find a tiny, clinically meaningless difference to be "statistically significant" (e.g., a weight loss drug that leads to an average loss of 200g over a year might have p < 0.001 but is clinically useless).

This is why we need a more informative measure: the Confidence Interval.

3. The Confidence Interval (CI): A Range of Plausible Values

A Confidence Interval gives us a range of values within which the true effect in the population likely lies. It provides two crucial pieces of information that the p-value alone does not: the magnitude of the effect and the precision of the estimate.

• Magnitude: The numbers in the interval tell you how large or small the effect might be (e.g., a 2 to 10 kg weight loss).
• Precision: The width of the interval tells you how certain we are about the effect. A narrow CI (e.g., 5 to 6 kg weight loss) means we have a precise estimate. A wide CI (e.g., 1 to 20 kg weight loss) indicates a lot of uncertainty.

Interpreting the CI in Practice

The most common CI is the 95% CI, which is mathematically linked to the p-value threshold of 0.05. Here’s how to use it to assess statistical significance:

1. For comparisons of means (e.g., t-test): The null hypothesis is a difference of zero. If the 95% CI for the difference between two groups includes the value 0, the result is not statistically significant (p > 0.05).
2. For ratios (e.g., Risk Ratio, Odds Ratio): The null hypothesis is a ratio of 1.0 (meaning equal risk in both groups). If the 95% CI for the ratio includes the value 1.0, the result is not statistically significant (p > 0.05).

To dive deeper into this and solidify the distinction between statistical and clinical relevance, the following article is an essential read for clinicians.

4. Integrating Statistical and Clinical Significance

Now, let's combine these concepts. When you read a study, you must ask two questions:

1. Is the result statistically significant? (Look at the p-value or whether the CI crosses the null value).
2. If so, is the result clinically significant? (Look at the values within the CI and use your clinical judgment to decide if the effect size is meaningful).

This image provides a powerful visual framework for this dual assessment. It introduces a "range of clinical indifference"—a range of effects that you would consider too small to be clinically meaningful.

Let's analyze the four scenarios in a Swasthawritta context, imagining a study on an intervention to reduce systolic blood pressure (SBP):

• Scenario A: The CI includes 0 (the null value). The result is not statistically significant. We don't have enough evidence to conclude the intervention works.
• Scenario B: The entire CI is on the beneficial side of 0, but it falls entirely within the "range of clinical indifference" (e.g., the CI for SBP reduction is 0.5 to 1.5 mmHg). The result is statistically significant, but not clinically significant. The effect is real, but too small to matter.
• Scenario C: The CI includes 0, so the result is not statistically significant. However, the CI is very wide and includes clinically important values (e.g., a reduction of 15 mmHg). This suggests the study was underpowered (too small) and inconclusive. We can't rule out a clinically important benefit.
• Scenario D: The entire CI is on the beneficial side of 0 and also falls outside the range of clinical indifference (e.g., the CI for SBP reduction is 5 to 15 mmHg). This is the ideal result: both statistically and clinically significant.

5. Advanced Interpretation: The "Compatibility" Approach

As you've seen, relying on a strict p<0.05 cutoff can be misleading. Many leading statisticians now advocate for a more nuanced approach, reframing p-values and CIs as measures of compatibility.

This perspective encourages you to think about which effect sizes are most compatible with your data, rather than making a simple yes/no decision.

The key idea is that a 95% CI simply shows you all the possible effect sizes that are reasonably compatible with your data (i.e., all the hypotheses that would give p > 0.05). This moves you away from black-and-white thinking and towards a more holistic assessment of the evidence.

6. Application: Interpreting a Real Clinical Trial

Let's finish by seeing how these concepts are applied in the complex world of a real clinical trial analysis. The following video discusses several landmark trials, and the speaker's thought process is highly instructive.

Conclusion

In this lesson, we have demystified the core statistical outputs you will encounter when reading research. We have moved from a simplistic view of p-values to a more robust, clinically-oriented interpretation that integrates effect size and precision through confidence intervals.

Key Takeaways:

• The p-value tells you the probability of seeing your data (or more extreme data) if there were truly no effect. A small p-value (typically ≤ 0.05) suggests the effect is unlikely to be due to chance.
• The p-value does not measure the size or importance of an effect.
• The 95% Confidence Interval (CI) provides a range of plausible values for the true effect size in the population. It communicates both the magnitude and precision of the finding.
• The most critical task for a clinician is to distinguish statistical significance (Is the effect real?) from clinical significance (Is the effect large enough to matter?). Use the CI to make this judgment.
• Thinking in terms of compatibility rather than a simple significant/non-significant dichotomy leads to more nuanced and appropriate clinical conclusions.

Preview of the Next Lesson:

Now that you can interpret the statistical heart of a research paper, you are ready for the next step: systematically appraising the entire study for its quality. A result can be statistically significant, but if the study was poorly designed or conducted, the finding may be invalid due to bias. In our next lesson, "Select and apply a structured critical appraisal tool (e.g., CASP checklist) to a research article on a Swasthawritta practice," we will learn how to formally assess a study's validity.