Exact Tests for Small Expected Frequencies

The Problem with Small Samples

The chi-square test works by comparing the observed counts in your contingency table to the counts you would expect if there were no association. The formula for the chi-square statistic, ${\chi }^2=\sum \frac{(O-E{)}^2}{E}$, produces a value that, for large samples, closely follows a theoretical chi-square distribution.

The problem arises when samples are small. If the expected count (E) in any cell is very low, that cell can have a disproportionately large effect on the total ${\chi }^2$ statistic, making the approximation to the theoretical chi-square distribution unreliable. The p-value you calculate could be wrong, leading to an incorrect conclusion.

To avoid this, we need a clear rule for when the approximation is no longer good enough.

The Expected Frequency Rule

The decision to use a chi-square test or an exact test hinges on the expected cell frequencies. As a quick reminder, you calculate the expected frequency for any cell using the formula:

The most common guideline, often called "Cochran's rule," provides the criteria for when the chi-square test is valid.

To start, let's hear this rule stated clearly.

As the video states, the core issue is an expected count being too low. Here are the specific rules of thumb you should know for your exam:

• For 2x2 tables: If any expected cell frequency is less than 5, the chi-square test is not reliable. You must use Fisher's exact test.

• For larger tables (r x c): The rule is slightly more lenient. The chi-square test is considered valid if:

  1. No more than 20% of the cells have an expected frequency less than 5.
  2. No cell has an expected frequency of less than 1.
     If these conditions are not met, you should consider using an exact test or combining categories to increase the expected counts.

The most common error in applying the chi-square test is failing to check this assumption first. The image below shows a typical output from statistical software, which often includes a note explicitly checking this condition.

A Complete Decision Framework

Now, let's put this rule into a step-by-step process. The following reading provides a fantastic guide for deciding between the chi-square and Fisher's exact tests, complete with a worked example where the rule is violated.

The example from the reading is critical. It shows that in borderline cases, the chi-square test can be anti-conservative, meaning it produces a smaller p-value and might lead you to incorrectly reject the null hypothesis. This is precisely why the rule exists and why using the correct test is essential.

Decision Flowchart and Common Mistakes

To help you internalize this process for the exam, let's review a decision flowchart. This provides a clear, strategic path for analyzing a contingency table.

The resource you just read, LINK, contains an excellent flowchart. Let's break down its logic:

Step 1: Are both variables categorical?

• If not, these tests don't apply.

Step 2: What is the table dimension?

• This determines which specific rule to apply (2x2 vs. larger table).

Step 3 & 4: Check expected frequencies.

• This is our key step. Calculate $E=(Row\text{ Total}\times \text{Column Total})/\text{Grand Total}$ for each cell.
• For a 2x2 table: Is any expected frequency < 5? If yes, use Fisher's. If no, use Chi-Square.
• For a larger table: Are more than 20% of expected frequencies < 5? If yes, use Fisher's or merge categories.

The resource also highlights common mistakes. For your exam, be especially mindful of these two:

1. Using Chi-Square with Small Expected Frequencies: This is the #1 error. Always check your expected counts first!
2. Confusing Observed and Expected Frequencies: The rule applies to the calculated expected counts, not the actual data you observed. An observed count of 0 is fine as long as the expected count is high enough.

Practice Problem

A psychologist is studying the effect of a new therapy on a rare phobia. They recruit 18 patients for a small pilot study.

Your Task:

1. Calculate the expected frequency for each of the four cells in the table.
2. Based on your calculations, is the Pearson chi-square test an appropriate method for analyzing this data? Justify your answer.

Take a few minutes to calculate the expected frequencies and make your decision before checking the solution.

Conclusion

This lesson focused on a crucial checkpoint in categorical data analysis. Before you interpret a chi-square statistic, you must validate its primary assumption.

Key Takeaways:

• The chi-square test relies on an approximation that is only valid for sufficiently large samples.
• The validity is checked using expected frequencies, not observed frequencies.
• The Rule: For a 2x2 table, if any expected count is less than 5, you must use Fisher's exact test. For larger tables, if more than 20% of expected counts are less than 5, the chi-square test is also unreliable.
• Your Exam Strategy: For any contingency table problem, your first step after setting up the table should always be to calculate the expected frequencies to decide which test is appropriate.

Now that we know exactly when to use an exact test, our next lesson will dive into the mechanics of Fisher's exact test. We will learn how to calculate the probability of our observed table and any more "extreme" tables to find the exact p-value.