One-Way ANOVA and Post-Hoc Tests

1. The Logic of ANOVA: Analyzing Variances to Compare Means

The central challenge when comparing more than two groups is controlling the family-wise error rate—the probability of making at least one Type I error (a false positive) across all the comparisons. If you run three separate t-tests, each at a 0.05 significance level, your chance of getting at least one false positive is actually closer to 14%, not 5%. As the number of groups increases, this problem gets much worse.

ANOVA cleverly solves this by asking a single question: is the variation between the groups significantly greater than the variation within the groups?

• Variation Between Groups ($S{S}_b$): This measures how much the mean of each group differs from the overall mean of all data combined. If the treatments have different effects, the group means will be far apart, and this variation will be large.
• Variation Within Groups ($S{S}_w$): This measures the random variation of individual data points around their own group's mean. This is often considered the "noise" or "error" in the data.

ANOVA calculates a test statistic called the F-ratio, which is the ratio of the between-group variance to the within-group variance.

If the F-ratio is large, it suggests that the differences between the group means are substantial compared to the random noise, leading us to reject the null hypothesis.

The hypotheses for a one-way ANOVA are:

• Null Hypothesis ($H_0$): The means of all groups are equal (${\mu }_1={\mu }_2=...={\mu }_k$).
• Alternative Hypothesis ($H_1$): At least one group mean is different from the others.

To get a formal understanding of how this partitioning of variance works, the following reading is an excellent, in-depth explanation.

2. Performing a One-Way ANOVA in R

Now, let's move from theory to practice. We will use R to perform a one-way ANOVA. The primary function for this is aov() (Analysis Of Variance). It uses the familiar formula syntax outcome ~ group, where outcome is your continuous dependent variable and group is your categorical independent variable.

Let's use the clinical trial example from the reading. Assume the data is in a data frame called clin.trial, with mood.gain as the outcome and drug as the grouping variable (with levels: 'placebo', 'anxifree', 'joyzepam').

Step 1: Run the ANOVA and Get the Summary

First, we fit the ANOVA model and store it in an object. Then, we use the summary() function to view the results.

# If you were following along with the book, you would have loaded the data.
# For now, let's assume `clin.trial` exists.




# Fit the ANOVA model
anova_model <- aov(mood.gain ~ drug, data = clin.trial)




# Get the ANOVA table
summary(anova_model)

The output will be a classic ANOVA table, just like the one you saw in the reading:

##             Df Sum Sq Mean Sq F value   Pr(>F)    
## drug         2  3.453   1.727   18.62 8.67e-05 ***
## Residuals   15  1.392   0.093                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Interpretation:

• Df: Degrees of freedom for the groups (drug, k-1 = 2) and the residuals (N-k = 15).
• Sum Sq: The between-group (drug) and within-group (Residuals) sums of squares.
• Mean Sq: The mean squares (Sum Sq / Df). This is the variance.
• F value: The F-ratio (18.62).
• Pr(>F): The p-value (8.67e-05).

Since the p-value is much smaller than 0.05, we reject the null hypothesis. We conclude that there is a statistically significant difference in mood gain among the three drug groups.

This video provides a concise end-to-end walkthrough of this entire process, from visualization to running the aov() command.

3. Post-Hoc Tests: Finding Where the Differences Lie

The significant p-value from our ANOVA is an "omnibus" result—it tells us that at least one group is different, but not which one. Did Joyzepam work better than the placebo? Was it also better than Anxifree? To answer these specific questions, we need to perform post-hoc tests.

A very common and robust post-hoc test is Tukey's Honestly Significant Difference (HSD) test. It performs all possible pairwise comparisons between the group means while controlling the family-wise error rate at our specified level (e.g., 5%).

Performing and Interpreting Tukey's HSD

In R, you can run this test directly on your aov model object using the TukeyHSD() function.

# Perform Tukey's HSD test
tukey_results <- TukeyHSD(anova_model)




# Print the results
print(tukey_results)

The output will show a table of pairwise comparisons:

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = mood.gain ~ drug, data = clin.trial)
## 
## $drug
##                        diff         lwr       upr     p adj
## anxifree-placebo  0.2666667 -0.15538584 0.6887192 0.2520970
## joyzepam-placebo  1.0333333  0.61128080 1.4553858 0.0000898
## joyzepam-anxifree 0.7666667  0.34461416 1.1887192 0.0009612

Interpretation:

• diff: The difference in means between the two groups being compared.
• lwr & upr: The lower and upper bounds of the 95% confidence interval for the difference.
• p adj: The adjusted p-value for that specific comparison.

Look at the p adj column. We see that the difference between joyzepam-placebo (p < .001) and joyzepam-anxifree (p < .01) are both statistically significant. The difference between anxifree-placebo (p = 0.25) is not.

An alternative way to interpret this is to look at the confidence intervals (lwr, upr). If the interval does not contain zero, the difference is statistically significant.

• anxifree-placebo: The interval (-0.16 to 0.69) contains 0. Not significant.
• joyzepam-placebo: The interval (0.61 to 1.46) does not contain 0. Significant.
• joyzepam-anxifree: The interval (0.34 to 1.19) does not contain 0. Significant.

This leads to a clear conclusion: Joyzepam led to a significantly greater mood gain than both the placebo and Anxifree. There was no significant difference between Anxifree and the placebo.

This plot visually represents the output of a Tukey HSD test.

Tidying Up with broom

As you build reproducible analysis pipelines, you'll want to store results in clean, predictable data frames. This is where the broom package shines. You first used it with t.test() and chisq.test(), and it's invaluable for ANOVA as well.

You can apply broom::tidy() to both the aov object and the TukeyHSD object.

# Make sure you have the packages installed and loaded
# install.packages("broom")
library(broom)




# Tidy the ANOVA results
tidy(anova_model)




# Tidy the Tukey HSD results
tidy(tukey_results)

The output of tidy(tukey_results) is particularly useful for reporting or plotting, as it gives you a clean table with one row per comparison.

## # A tibble: 3 × 7
##   term  contrast              null.value  estimate conf.low conf.high adj.p.value
##   <chr> <chr>                      <dbl>     <dbl>    <dbl>     <dbl>       <dbl>
## 1 drug  anxifree-placebo               0     0.267   -0.155     0.689   0.252    
## 2 drug  joyzepam-placebo               0     1.03     0.611     1.46    0.0000898
## 3 drug  joyzepam-anxifree              0     0.767    0.345     1.19    0.000961 

This "tidy" approach is central to modern data analysis in R. To learn more about how broom streamlines this process, this guide is excellent.

4. Assumptions and Alternatives

Like the t-test, one-way ANOVA relies on three main assumptions:

1. Independence: The observations in each group are independent of each other. This is usually handled by your study design (e.g., random assignment of participants).
2. Normality: The residuals of the model (the within-group errors) are approximately normally distributed.
3. Homogeneity of variance (homoscedasticity): The variance of the residuals is equal across all groups.

ANOVA is fairly robust to violations of normality and homogeneity, especially if your group sizes are equal and reasonably large. However, if these assumptions are severely violated, you should use a non-parametric alternative. For a one-way ANOVA, the appropriate non-parametric test is the Kruskal-Wallis test. It's essentially an ANOVA performed on the ranks of the data rather than the raw values.

You can perform it in R with the kruskal.test() function.

kruskal.test(mood.gain ~ drug, data = clin.trial)

The interpretation is similar: a small p-value suggests that at least one group is different. Post-hoc tests for a Kruskal-Wallis test also exist but are slightly more complex. For your work, focusing on meeting the assumptions for ANOVA is often the first and best step.

Conclusion

You have now mastered a cornerstone of statistical analysis. ANOVA allows you to move beyond simple two-group comparisons and rigorously test for differences across multiple groups, a frequent necessity in medical and clinical research.

Key Takeaways:

• One-way ANOVA is used to compare the means of a continuous variable across three or more categorical groups.
• It works by comparing the variance between groups to the variance within groups, summarized by the F-ratio.
• A significant ANOVA result (p < 0.05) is an "omnibus" test; it indicates a difference exists somewhere among the groups.
• Post-hoc tests, like Tukey's HSD, are required to perform pairwise comparisons and identify which specific groups differ, while controlling the overall Type I error rate.
• In R, the workflow is aov(), followed by summary(), and then TukeyHSD() on the model object for post-hoc analysis.
• The broom::tidy() function is essential for converting ANOVA and Tukey HSD results into clean data frames for reporting and further analysis.

Preview of the Next Lesson:
In this lesson, you've learned to run several key hypothesis tests. A crucial part of a professional workflow is presenting the results of these tests in a clean, consistent, and machine-readable format. Our next two lessons will focus squarely on the broom package, showing you how to systematically tidy() the outputs from all the tests you've learned (t.test, chisq.test, aov) and interpret the key columns it produces. This will set you up perfectly for automating reports and creating publication-quality tables.