Introduction
In our last lesson, we established a critical decision rule: when analyzing a 2x2 contingency table, if any expected cell frequency is less than 5, the chi-square test's p-value is unreliable. In these cases, you must use Fisher's exact test.
Today, we move from the when to the how. This lesson will walk you through the mechanics of performing Fisher's exact test. Your goal is to be able to take a 2x2 table, calculate the probability of the observed result, identify all "more extreme" results, and sum their probabilities to find the exact p-value. This is a very procedural skill, perfect for exam preparation.
The core idea is that we calculate the probability of our observed data directly, without relying on an approximation like the chi-square distribution. This calculation is rooted in combinatorics, a concept that may be familiar from your computer science background.
The Logic: Fixed Margins and Hypergeometric Probability
Imagine you have a bag of marbles corresponding to the totals in your study. For a study with 10 smokers and 15 non-smokers (column totals) and 8 people with cancer and 17 without (row totals), Fisher's test asks: "If I randomly draw 8 people for the 'cancer' group from the total pool of 25, what is the probability that exactly x of them would be smokers?"
This is the fundamental principle: we assume the row and column totals (the margins) are fixed. The test then calculates the probability of observing our specific cell counts, given those fixed margins. The probability distribution that governs this is the hypergeometric distribution.
To build a solid intuition for this, let's watch a short video that uses a classic "balls in a box" analogy.
Fisher's test and how to calculate the exact p-value
To understand how Fisher's test works, we first need to grasp how it calculates the probability of a specific table. The test is based on the hypergeometric distribution, which sounds complex but has an intuitive foundation in combinatorics. This video provides an excellent analogy to build this intuition.
Watch the segment from 02:16 to 07:12. Focus on how the problem of drawing colored balls from a box translates to a 2x2 contingency table. This will help you understand the formula we're about to use.
The Formula for an Exact Probability
As the video explained, we can calculate the exact probability of observing any specific 2x2 table configuration. For a generic table with cell counts a, b, c, d and total n:
| Category 1 | Category 2 | Total | |
|---|---|---|---|
| Group 1 | a | b | a+b |
| Group 2 | c | d | c+d |
| Total | a+c | b+d | n |
The probability of this specific arrangement occurring by chance (under the null hypothesis of no association), given the fixed margins, is calculated using the following formula:
Where ! denotes a factorial (e.g., ). This formula directly calculates a single point from the hypergeometric distribution.
Let's see this formula applied in a worked example.
The Fisher's Exact Test - Technology Networks
Now that you have the intuition, let's look at the formula and apply it. The following article provides a clear definition of the formula and a step-by-step example.
Please read the following sections from the article: Find the heading 'Interpreting Fisher’s test results'. Read the part that introduces the generic 2x2 table and the formula for the exact probability (P). Continue to the heading 'A Fisher exact test example'. Read up to 'Step three' to see how the contingency table is populated and how the exact probability for the observed table is calculated.
As the article demonstrates with the coffee and cancer data, you can plug the cell and marginal counts into the formula to find the exact probability of observing that specific table. For their observed table (a=3, b=11, c=1, d=15), the probability is 0.212.
From a Single Probability to a P-Value
The p-value isn't just the probability of our one observed table. It's the probability of observing a result at least as extreme as the one we found, assuming the null hypothesis is true. To find this, we must:
- Identify all possible tables that are "more extreme" than our observed table.
- Calculate the probability for each of these extreme tables.
- Sum these probabilities together.
"More extreme" means tables that show an even stronger association between the variables, while still maintaining the same fixed row and column totals.
This next video segment explains this process very clearly.
Fisher's test and how to calculate the exact p-value
The p-value isn't just the probability of our observed table. It's the probability of getting a result at least as extreme as what we observed. This video clearly explains how to find these 'more extreme' tables and use their probabilities to calculate the final p-value.
Watch from 07:12 to the end (13:00). Pay close attention to: How a 'more extreme' table is generated by shifting counts while keeping the totals fixed (10:07). How the one-sided p-value is the sum of the observed probability and the more extreme probability (11:04). The method for calculating the two-sided p-value: summing the probabilities of all tables that are less than or equal to the probability of the observed table (11:50).
Full Worked Example: Calculating a Two-Sided P-Value
Let's synthesize everything we've learned and perform a full analysis of the coffee/cancer example from the LINK article.
Step 1: List all possible tables and their probabilities.
The margins are fixed: Row totals (14, 16) and Column totals (4, 26). The count in cell 'a' (Coffee drinkers, Cancer) can range from 0 to 4. All other cells are determined once 'a' is set.
The LINK article lists all possible tables and their probabilities:
- Table (a=0): P = 0.066
- Table (a=1): P = 0.286
- Table (a=2): P = 0.398
- Table (a=3): This is our observed table. P = 0.212
- Table (a=4): P = 0.036
Step 2: Define "as extreme or more extreme".
We will calculate a two-sided p-value, which tests for any association (positive or negative). As the video explained, the standard method used by statistical software is to sum the probabilities of all tables that are equally likely or less likely than the observed table.
Our observed table has a probability of 0.212.
Step 3: Identify tables with P ≤ 0.212.
Looking at our list of probabilities:
- P(a=0) = 0.066 (≤ 0.212)
- P(a=1) = 0.286 (> 0.212)
- P(a=2) = 0.398 (> 0.212)
- P(a=3) = 0.212 (≤ 0.212, this is the observed table itself)
- P(a=4) = 0.036 (≤ 0.212)
The tables that are "as or more extreme" are the ones with a=0, a=3, and a=4.
Step 4: Sum the probabilities to get the two-sided p-value.
Conclusion: Since the p-value (0.314) is greater than the typical significance level of 0.05, we fail to reject the null hypothesis. We conclude there is no significant evidence of an association between coffee drinking and cancer in this study.
(Note: The LINK article arrives at a different p-value due to a likely typo in its summation. The method and calculation shown here are the standard approach and align with the video's explanation.)
Practice Problem
Let's return to the practice problem from our last lesson. A psychologist studies a new therapy for a rare phobia.
| Therapy Group | Phobia Improved | Phobia Not Improved | Row Total |
|---|---|---|---|
| New Therapy | 7 | 2 | 9 |
| Standard Care | 4 | 5 | 9 |
| Column Total | 11 | 7 | 18 |
We already determined that because the expected counts were low, Fisher's exact test is required. Now, let's perform the test.
Your Task:
- Our observed table has a=7. The two tables that are more extreme in either direction are when a=8 and a=9. The probabilities for the three most extreme tables are:
- P(a=7) [Observed] = 0.170
- P(a=8) = 0.032
- P(a=9) = 0.002
- Using these probabilities, calculate the one-sided p-value. Assume the alternative hypothesis is that the new therapy is more effective (i.e., a right-tailed test).
- For a two-sided test, you would need the probabilities of all possible tables. Given just the information above, which tables would definitely be included in the two-sided p-value calculation?
Click for Solution
-
One-sided p-value:
A one-sided test for improvement includes the probability of the observed result plus any results that are more extreme in the same direction. Here, "more extreme" means an even stronger positive effect of the new therapy (a=8 and a=9).Since 0.204 > 0.05, we would conclude there is not a statistically significant improvement with the new therapy.
-
Tables for the two-sided p-value:
For a two-sided test, we sum the probabilities of all tables with a probability less than or equal to the observed table's probability (P ≤ 0.170). Based on the information given:- P(a=7) = 0.170 (This is included as it is equal to our threshold).
- P(a=8) = 0.032 (This is included as 0.032 < 0.170).
- P(a=9) = 0.002 (This is included as 0.002 < 0.170).
We would also need to calculate the probabilities for tables where a < 7 and include any of those where P ≤ 0.170.
Conclusion
You have now learned the complete mechanical process for Fisher's exact test. You can apply this directly to any 2x2 table you encounter on your exam where the chi-square test assumptions are not met.
Key Takeaways:
- Fisher's exact test calculates probability directly from the hypergeometric distribution, assuming fixed row and column margins.
- The first step is to calculate the probability of the observed table using the factorial formula.
- The p-value is the sum of probabilities of all tables that are as extreme or more extreme than the observed one.
- For a two-sided test, this means summing the probabilities of all tables that are equally or less likely to occur than your observed table.
In the next lesson, we will move to larger r x c tables and discuss the distinction between the generalized Fisher's exact test and a different approach called the permutation chi-square test.