Hypothesis Testing with Categorical Data

1. The Pearson Chi-Square Test in Practice

Let's start with the most common test for independence: the Pearson chi-square test. As a quick refresher, the test statistic, $X^2$, quantifies the total difference between the observed counts in your table and the counts you would expect if the null hypothesis of no association were true.

Now, let's see how to apply this to a real dataset.

An Example: Mites and Plant Disease

We'll use an example from one of our resources involving an experiment on cotton plants. Researchers wanted to know if a prior infestation of spider mites would induce resistance to a fungal disease called Wilt.

The data is summarized in this 2x2 contingency table:

The research question is: Is there a statistically significant association between mite treatment and the incidence of Wilt disease?

• Null Hypothesis (H₀): Mite treatment and the incidence of Wilt are independent. The proportion of plants that get Wilt is the same for both the treated and untreated groups.
• Alternative Hypothesis (Hₐ): Mite treatment and the incidence of Wilt are not independent.

Applying the Test

Instead of calculating the $X^2$ statistic by hand, we use software. The following resource shows how this is done in R and provides the output we need to interpret.

Interpreting and Reporting the Results

As you saw in the reading, the R command chisq.test(table, correct=FALSE) gives us the key pieces of information. For the Mites data, the output is: X-squared = 7.2037, df = 1, p-value = 0.007275.

Here's how to break that down:

• X-squared = 7.2037: This is the calculated chi-square statistic. It's a measure of how far our observed data deviates from the null hypothesis. A larger value suggests a greater deviation.
• df = 1: These are the degrees of freedom. For a contingency table, this is calculated as $(Rows-1)\times (Columns-1)$. For our 2x2 table, it's $(2-1)\times (2-1)=1$. The degrees of freedom determine the specific shape of the chi-square distribution we compare our statistic against.
• p-value = 0.007275: This is the probability of observing a chi-square statistic as large as 7.2037 (or larger) if the null hypothesis were true.

Since our p-value (0.0073) is less than the conventional significance level of $\alpha =0.05$, we reject the null hypothesis.

How to Report the Finding:

When reporting your results, you should always include the test statistic, degrees of freedom, p-value, and a clear statement of your conclusion in the context of the problem.

A Pearson's chi-square test was conducted to examine the relationship between mite treatment and the incidence of Wilt disease. The results indicated a significant association between the two variables, ${\chi }^2(1)=7.20$, $p=.007$. Plants that were not treated with mites were more likely to develop Wilt disease compared to those that were.

2. When Assumptions Fail: Fisher's and Permutation Tests

The chi-square test is an approximation that works well when sample sizes are large. Its primary assumption is that all expected cell counts should be greater than 5. When this assumption is violated, the calculated p-value may not be accurate.

This is a critical point for your functional understanding: you must always check the expected counts before trusting a chi-square result.

When you encounter this situation, you have two excellent alternatives.

2.1. Fisher's Exact Test: The Gold Standard for 2x2 Tables

As we've discussed, Fisher's Exact Test calculates the exact probability of observing the data, so it doesn't rely on the large-sample approximation. It is the preferred method for 2x2 tables with small expected counts.

Applying the Test:

Let's look at the resource below to see how to perform Fisher's test.

Interpreting and Reporting:

The output of fisher.test() gives you a p-value directly. For example, if we ran it on a dataset and got p-value = 0.021, we would conclude there is a significant association. The output also provides an odds ratio, a measure of association we studied earlier, which is very useful for interpretation.

How to Report the Finding:

Due to small expected cell counts, Fisher's exact test was used to examine the relationship between Variable A and Variable B. The test revealed a significant association between the variables (p = .021).

2.2. The Permutation Test: A Flexible Alternative

What if your table is larger than 2x2 and you have small expected counts? Fisher's test becomes computationally complex for large tables. This is where the permutation test we learned about in the last lesson becomes extremely useful. It provides a robust p-value without relying on the chi-square distribution's assumptions.

Applying the Test:

Fortunately, performing a permutation test in R is very simple. We use the same chisq.test function but add an argument to tell it to simulate the p-value by shuffling the data.

Interpreting and Reporting:

The output will look similar to the standard test, but the p-value is derived from thousands of random shuffles. For the Mites data, the simulated p-value was 0.0148.

How to Report the Finding:

A chi-square test of independence with a simulated p-value based on 10,000 permutations was conducted... The result was significant (${\chi }^2=7.20$, simulated p = .015), indicating an association between mite treatment and Wilt disease.

3. A Decision Framework: Choosing the Right Test

You now have three tools. Here is a simple framework to help you choose the right one every time.

1. Construct your contingency table and calculate the expected counts for each cell.

   • Expected Count = $\frac{(\text{Row Total})\times (\text{Column Total})}{\text{Grand Total}}$

2. Check the expected counts against the > 5 rule.

   • If all expected counts are > 5:

     • The Pearson chi-square test is reliable and appropriate. Use chisq.test().

   • If one or more expected counts are ≤ 5:

     • Is the table 2x2?
       • Yes -> Use Fisher's Exact Test. It is the most powerful and exact method in this scenario. Use fisher.test().
     • Is the table larger than 2x2?
       • Yes -> Use the chi-square test with permutation. It provides a reliable p-value when the theoretical distribution is not a good fit. Use chisq.test(..., simulate.p.value=TRUE).

This framework ensures your analysis is always statistically sound.

4. A Complete Example: R x C Table

Let's apply this framework to a larger table. The following example investigates the relationship between race and marital status using survey data. This is a 2x4 table.

In this example, all expected cell counts were well above 5, so the standard Pearson's chi-square test was the correct choice. The test yielded ${\chi }^2(3)=26.168$ and a very small p-value ($p<.001$), leading to the conclusion that there is a significant association between race and marital status in this dataset. If the expected counts had been small, the correct procedure would have been to run chisq.test(tab, simulate.p.value=TRUE).

Conclusion

In this lesson, we made the critical transition from theory to practice. You learned not just how to perform tests of independence, but more importantly, when to use each one.

Key Takeaways:

• Standard Procedure: The Pearson chi-square test is the default for testing independence in contingency tables.
• The Critical Assumption: Always check that all expected cell counts are greater than 5.
• Your Toolkit for Violations:
  • For 2x2 tables with small expected counts, use Fisher's Exact Test.
  • For R x C tables with small expected counts, use a permutation-based chi-square test.
• Reporting: Always report the test used, the test statistic value, degrees of freedom (if applicable), the p-value, and your conclusion in the context of the data.

Preview of the Next Lesson:

So far, we have treated all categorical variables as nominal, meaning the categories have no intrinsic order (e.g., 'Mites' vs. 'No Mites'). In our next lesson, we will explore what happens when our variables are ordinal—when the categories have a natural ranking (e.g., 'low', 'medium', 'high'). We will learn about specialized tests, like the Cochran-Armitage trend test, which are more powerful for detecting ordered associations.