Understanding Multiple Comparisons and Bonferroni Correction

The Problem: Finding "Significance" by Chance

When we set our significance level $\alpha$ to 0.05, we are accepting a 5% chance of making a Type I error—that is, a 5% chance of rejecting the null hypothesis when it's actually true (a "false positive"). This is the risk we tolerate for a single test. But what happens when we run many tests? The probability of getting at least one false positive across the entire "family" of tests starts to climb dramatically.

This short video from the New England Journal of Medicine provides an excellent and memorable illustration of this very problem.

As the video highlights, this phenomenon is called alpha inflation. Each test is like buying a lottery ticket with a 5% chance of winning (a false positive). If you buy only one ticket, your chances are low. But if you buy 20, you're much more likely to have a "winner" by sheer luck.

Quantifying the Problem: The Family-Wise Error Rate (FWER)

To formalize this, statisticians talk about the Family-Wise Error Rate (FWER). This is the probability of making at least one Type I error in a family of hypothesis tests. If the tests are independent, the FWER can be calculated.

Let's use the formula you just read about. If we run 10 independent tests with $\alpha =0.05$, the probability of making at least one Type I error isn't 5%. It's:

So, you have a staggering 40% chance of finding at least one "significant" result just by chance! The graph below visualizes this explosive growth of the FWER as the number of tests increases, demonstrating why this is such a critical issue in research.

A Simple Solution: The Bonferroni Correction

How do we combat this? The most straightforward approach is the Bonferroni correction. The logic is simple and elegant: if you are going to perform $N$ tests, you protect your overall FWER by making the significance level for each individual test more stringent.

Specifically, you define a new, adjusted alpha for each test:

Where ${\alpha }_{overall}$ is your desired family-wise error rate (usually 0.05) and $N$ is the number of tests. If you perform 10 tests, your new significance threshold for each test becomes $0.05/10=0.005$. Only p-values below this much lower threshold are considered statistically significant.

This video provides a step-by-step walkthrough of this logic in the context of post-hoc tests after an ANOVA, a perfect real-world scenario.

Applying Bonferroni Correction in R

Instead of manually calculating a new alpha level, the standard practice in R is to calculate adjusted p-values. An adjusted p-value represents the smallest family-wise alpha at which a given test would be significant. The workflow is simple: you run your tests, get your original ("raw") p-values, and then compare the adjusted p-values directly to your original alpha (0.05).

The p.adjust() function in base R is the tool for this job. It takes a vector of p-values and the name of the correction method you want to use.

Let's see how this works within the tidyverse workflow you've learned. Imagine you have performed three t-tests to compare a continuous variable across three groups and have your results in a tidy data frame.

# A tibble of hypothetical t-test results
test_results <- tibble::tribble(
  ~term,          ~p.value,
  "Group A vs B", 0.041,
  "Group A vs C", 0.002,
  "Group B vs C", 0.150
)




# Add a new column with Bonferroni-adjusted p-values
library(dplyr)
test_results %>%
  mutate(p.adj = p.adjust(p.value, method = "bonferroni"))




# # A tibble: 3 × 3
#   term           p.value p.adj
#   <chr>            <dbl> <dbl>
# 1 Group A vs B     0.041 0.123
# 2 Group A vs C     0.002 0.006
# 3 Group B vs C     0.15  0.45 

Notice the results:

• The comparison "Group A vs B" had a raw p-value of 0.041, which was significant at $\alpha =0.05$. However, its adjusted p-value is 0.123. Since $0.123>0.05$, we now conclude this difference is not statistically significant after correcting for multiple comparisons.
• The comparison "Group A vs C" had a very small raw p-value (0.002). Its adjusted p-value is 0.006. Since $0.006<0.05$, this result remains statistically significant.

This demonstrates how Bonferroni correction helps you filter out results that might have appeared significant only due to alpha inflation. The following resource shows this exact syntax in the context of a linear model, confirming its broad applicability.

The Trade-off: Conservativeness and Statistical Power

The Bonferroni correction is popular because it is simple to understand and guarantees control of the FWER. However, it is known for being conservative. By making the significance threshold so strict, it reduces the chance of Type I errors (false positives) but increases the chance of Type II errors (false negatives)—that is, you might fail to detect a real effect. This is the classic trade-off between sensitivity and specificity that is ubiquitous in medicine and statistics.

Bonferroni is not the only correction method. Others, like Holm-Bonferroni or Benjamini-Hochberg (which controls the "False Discovery Rate" instead of the FWER), are more powerful (less conservative) under certain conditions.

As the plot shows, the Bonferroni method adjusts p-values more aggressively than others. The choice of method depends on your research context. In exploratory analyses (like genomics, where you test thousands of genes), less conservative methods are common. In confirmatory clinical trials with a few key secondary endpoints, the rigorous control of Bonferroni might be preferred to avoid making false claims about a drug's efficacy.

Conclusion

In this lesson, you have confronted a fundamental challenge in statistical analysis: the problem of multiple comparisons. You learned that running many tests inflates the probability of finding a "significant" result by chance alone.

Key Takeaways:

• Performing multiple hypothesis tests inflates the Family-Wise Error Rate (FWER), the probability of making at least one Type I error.
• This alpha inflation can lead to spurious findings and is a critical issue to address in any analysis involving more than one statistical test.
• The Bonferroni correction is a simple and common method to control the FWER by adjusting the significance threshold for each test to $\alpha /N$.
• In R, this is practically applied by using p.adjust(p.value, method = "bonferroni") to calculate adjusted p-values, which are then compared to the original alpha of 0.05.
• The Bonferroni method is effective but conservative, meaning it reduces statistical power and increases the risk of false negatives (Type II errors).

Preview of the Next Lesson:
This lesson concludes our module on the foundations of hypothesis testing. You now have a robust toolkit for performing, tidying, interpreting, and correcting the results of common statistical tests. In the next module, we will begin a deep dive into one of the most powerful and widely used statistical techniques: regression modeling. You will start by learning how to fit and interpret a simple linear regression model using the lm() function and how to diagnose whether the model is a good fit for your data.