Tukey's HSD for Lingerie Style Comparisons

The Problem with Multiple Comparisons

After getting a significant ANOVA result, a natural impulse might be to run a series of t-tests for every possible pair of styles. For instance, with four bottom styles (Thong, Brazilian, Boyshort, High-Waist), you would need to run six separate t-tests:

• Thong vs. Brazilian
• Thong vs. Boyshort
• Thong vs. High-Waist
• Brazilian vs. Boyshort
• Brazilian vs. High-Waist
• Boyshort vs. High-Waist

While this approach seems straightforward, it introduces a serious statistical problem: the inflation of the Type I error rate. As you know from your background, if you set your significance level ($\alpha$) at 0.05 for a single test, you accept a 5% chance of a false positive. However, when you run multiple tests, the probability of getting at least one false positive across the entire "family" of tests skyrockets. For six comparisons, the family-wise error rate (FWER) becomes:

This means you would have a 26.5% chance of incorrectly concluding that a difference exists when it doesn't—a risk far too high for making reliable design decisions. Post-hoc tests are designed specifically to solve this problem by allowing multiple comparisons while keeping the family-wise error rate at your desired level (e.g., 0.05).

The following resource from "Statistics by Jim" provides an excellent explanation of this concept.

Pinpointing the Differences with Tukey's HSD

Tukey's HSD is one of the most common post-hoc tests used after a significant one-way ANOVA. It compares all possible pairs of means simultaneously while controlling the family-wise error rate. The test works by calculating a single critical value, the "Honestly Significant Difference," which acts as a yardstick. If the absolute difference between any two sample means is greater than this HSD value, we conclude that the difference is statistically significant.

The "Yardstick" Calculation

The formula for the HSD value is:

Where:

• $q$ is the critical value from the studentized range distribution. You find this value in a table using your significance level ($\alpha$), the number of groups ($k$), and the degrees of freedom for the error term ($d{f}_{error}$) from your ANOVA table.
• $M{S}_{error}$ is the Mean Square Error (or Within-Group Mean Square) from your ANOVA table. It's our pooled estimate of the variance, or "noise."
• $n$ is the sample size per group (assuming a balanced design).

This video from statslectures provides a clear, step-by-step example of calculating and applying this value.

Interpretation with Confidence Intervals

Modern statistical software often presents Tukey's HSD results as a set of simultaneous confidence intervals for the differences between means. This is a particularly intuitive way to interpret the results. For each pair of styles (e.g., Push-up vs. Plunge), the output provides a 95% confidence interval for the true difference in their mean appeal scores.

The rule for interpretation is simple:

• If a confidence interval does not contain zero, the difference between that pair of means is statistically significant.
• If a confidence interval does contain zero, the difference is not statistically significant.

This graphical representation is very powerful. Let's examine an example. Suppose we tested four materials (A, B, C, D) for strength. After a significant ANOVA, we run a Tukey test.

In this plot, we can quickly see:

• The intervals for (B-A), (C-A), (D-A), (C-B), and (D-C) all cross the zero line. This means we cannot conclude that these pairs are significantly different from each other.
• The interval for (D-B) is entirely to the right of zero. This indicates that the mean of group D is significantly higher than the mean of group B.

This visual method allows for immediate identification of significant results and also gives you a sense of the precision of the estimate (the width of the interval).

Practice Exercise

Let's return to the exercise from our previous lesson. You conducted an experiment comparing four lingerie bottom styles: thong, brazilian, boyshort, and high-waist. Your ANOVA produced a significant result ($p=0.02$), confirming a difference exists.

Here is the relevant information from your ANOVA table and your data collection:

• Number of groups ($k$): 4
• Sample size per group ($n$): 20
• Mean Square Error ($M{S}_{error}$): 20.0
• Degrees of Freedom for Error ($d{f}_{error}$): 76

After calculating the mean appeal scores for each style, you found:

• Thong: 9.5
• Brazilian: 8.8
• Boyshort: 5.2
• High-Waist: 5.0

Your Task:

1. Find the critical value for HSD. Using $\alpha =0.05$, $k=4$, and $d{f}_{error}=76$, the appropriate studentized range value is $q\approx 3.73$.
2. Calculate the HSD "yardstick" value. Use the formula provided above.
3. Perform all six pairwise comparisons. For each pair, calculate the absolute difference between their means.
4. Determine which pairs are significant. Compare each difference to your calculated HSD value.
5. Summarize your findings. Write a concluding paragraph explaining which styles are statistically superior for the photoshoot. You can use the common lettering notation (e.g., styles sharing a letter are not significantly different) to summarize the groups. For a detailed guide on this notation, you can refer to the Penn State resource below.

Take your time to work through the calculations and draw your final conclusions.

Conclusion

Today, you've added a crucial tool to your experimental design toolkit. You now understand not only why post-hoc tests are necessary but also how to conduct and interpret one of the most common methods, Tukey's HSD. By moving beyond the omnibus ANOVA F-test, you can now generate fine-grained, actionable insights from your fitting room experiments.

Key Takeaways:

• Running multiple t-tests after an ANOVA inflates the family-wise Type I error rate. Post-hoc tests like Tukey's HSD are designed to control this error.
• Tukey's HSD can be interpreted using a calculated critical value (the "yardstick") or by examining simultaneous confidence intervals for the differences between means.
• If the difference between two means exceeds the HSD value, or if the confidence interval for their difference does not contain zero, the two groups are considered statistically significantly different.

So far, our experiments have focused on a single factor, such as "bra style." But what if the appeal of a look depends on the combination of factors? For example, does a balconette bra work better with a thong, while a plunge bra works better with a brazilian cut? In our next lesson, we will explore two-way ANOVA, a powerful technique for analyzing the main effects of two factors and, crucially, their interaction.