Fisher's Exact vs. Permutation Chi-Square for R x C Tables

Introduction

In our last lesson, you mastered the mechanics of Fisher's exact test for 2x2 tables, a crucial tool when the chi-square test's assumptions about expected frequencies aren't met. We saw how it calculates an exact p-value by considering every possible table with the same marginal totals.

But what happens when our data isn't a simple 2x2 table? What if we have tables with more rows or columns (referred to as r x c tables)? The problem of low expected frequencies still exists, but the solution becomes more complex. This lesson tackles that challenge directly. We will explore two advanced methods for analyzing r x c tables with small samples: the generalized Fisher's exact test and the permutation chi-square test.

Your goal is to understand the fundamental difference in their approaches. This isn't about manual calculation, which is computationally prohibitive. Instead, you'll learn the conceptual logic behind each, their pros and cons, and the decision rules for when one might be used over the other—key knowledge for identifying the correct method on an exam.

The Challenge: Moving from 2x2 to r x c

When we move beyond a 2x2 table, the number of possible tables that share the same row and column totals explodes. For a 2x2 table, we could list all possibilities by hand. For a 3x3 table, this becomes tedious, and for a 4x5 table, it's virtually impossible to do manually.

This computational complexity presents a major hurdle for the "brute-force" method we used in the last lesson. Let's explore the two primary ways statisticians handle this.

Method 1: The Generalized Fisher's Exact Test

The most direct extension of what we've already learned is the generalized Fisher's exact test.

Core Idea: It operates on the exact same principle as the 2x2 test. It seeks to calculate the exact probability of every possible r x c table that could be formed with the observed row and column totals. The probability of any single table is calculated using a formula from the multivariate hypergeometric distribution (a generalization of the formula we used before).
P-Value Calculation: To get a p-value, it must sum the probabilities of all tables considered "as extreme or more extreme" than the one we observed.
The Problem: For a 2x2 table, "more extreme" was simple to define. For a larger r x c table, there is no single, obvious way to order the tables from least to most extreme. Different software packages might use different rules to do this, leading to slightly different p-values. More importantly, the sheer number of tables makes this process incredibly slow and memory-intensive.

Chi-square and Fisher's exact tests

The article we used previously to discuss the chi-square test briefly hints at this issue. Let's revisit it to see the practical advice offered and the mention of this generalization.

Find the section near the end that begins with 'The first objective of this article...'. Focus on two key sentences: 'All of the methods demonstrated for 2 × 2 tables can be generalized to r × c tables.' 'Yes, but it is more computationally expensive... If the chi-square assumptions are met, then the sample size is typically larger and these calculations become numerous.'\nThis confirms the existence of a generalized exact test and its primary drawback: computational cost.

Key Takeaway for the Exam: You need to know that the generalized Fisher's exact test is the "gold standard" in theory, as it provides an exact p-value without relying on distributional assumptions. However, its computational cost makes it impractical for all but the smallest r x c tables.

Method 2: The Permutation Chi-Square Test

Since calculating an exact p-value is often infeasible, we can use a clever simulation-based approach called a permutation test. This method should feel intuitive given your background in computer science, as it relies on resampling and iteration—concepts similar to Monte Carlo simulations.

Instead of calculating the probability of every table, a permutation test approximates the exact p-value by generating a large, random sample of possible tables.

Here is the algorithm:

Calculate Observed Statistic: For your actual, observed r x c table, calculate the standard Pearson chi-square statistic ( $χ^{2}$ ). Let's call this $χ_{obs}^{2}$ .
Pool and Shuffle: Imagine a dataset where each row is an individual. You have one column for their group (e.g., 'Treatment A', 'Treatment B', 'Control') and another column for their outcome (e.g., 'Improved', 'No Change', 'Worsened'). The core of the permutation test is to break the link between group and outcome. You do this by randomly shuffling—or permuting—the outcome labels across all individuals.
Create New Table and Statistic: Using the shuffled data, create a new r x c contingency table. Because the outcomes were randomly assigned, this new table represents a scenario consistent with the null hypothesis (no association). Calculate the chi-square statistic for this new permuted table, $χ_{perm}^{2}$ .
Repeat: Repeat steps 2 and 3 thousands of times (e.g., 10,000 or more). This process generates an empirical distribution of what the $χ^{2}$ statistic looks like when the null hypothesis is true, tailored specifically to your data's marginal totals.
Calculate P-Value: The p-value is simply the proportion of times your simulated $χ_{perm}^{2}$ values were greater than or equal to your original, observed $χ_{obs}^{2}$ .
$p -value = \frac{Number of permutations where χ _{perm}^{2} \geq χ _{obs}^{2}}{Total number of permutations}$

Why is this powerful? It doesn't rely on the assumption that the test statistic follows a theoretical chi-square distribution. Instead, it builds the distribution from your data. This makes it valid even when expected cell counts are low. It uses the $χ^{2}$ value not as a way to get a p-value from a theoretical curve, but simply as a score to rank how "extreme" a given table's association is.

The image below illustrates a key benefit of permutation tests: they can be more statistically powerful (better at detecting a real effect) than the standard chi-square test, especially when the latter's assumptions are not perfectly met.

Empirical Power Comparison: Permutation vs. Chi-squared Tests — This figure compares the empirical power of permutation tests (solid line) versus chi-squared tests (dashed line) across different scenarios. In each case, the permutation test demonstrates higher power, meaning it's more likely to correctly detect a true association, particularly as the sample size grows.

Head-to-Head: Fisher's Exact vs. Permutation Test

This is the core of the lesson. For your exam, you need to be able to distinguish these two approaches. Here is a summary table.

Feature	Generalized Fisher's Exact Test	Permutation Chi-Square Test
Test Type	Exact Test	Approximate Exact Test (uses simulation)
Method	Exhaustive enumeration of all possible tables.	Random sampling of possible tables via permutation.
Metric for "Extremeness"	The exact hypergeometric probability of the table itself.	The Pearson $χ^{2}$ statistic of the table.
P-Value Source	Summing exact probabilities of tables.	Proportion of simulated $χ^{2}$ statistics exceeding observed.
Computational Cost	Very high, often infeasible for `r x c` tables.	Moderate, feasible for most tables.
Primary Advantage	Theoretically perfect; guaranteed exact p-value.	Practical for larger tables; avoids chi-square assumptions.
Primary Disadvantage	Computationally impractical.	P-value is an approximation (though very accurate with enough permutations).

Decision Rule for the Exam:

If a question describes a situation with a small sample in an r x c table and asks for an exact test, it is referring to Fisher's exact test.
If a question describes a simulation or resampling procedure to generate a p-value for a contingency table, it is referring to a permutation test.
The permutation test is a practical compromise: it's more accurate than the standard chi-square test when assumptions are violated, and more feasible than the generalized Fisher's exact test.

Conclusion

You now understand the two sophisticated methods used for r x c contingency tables when the sample size is small. While you won't perform these complex calculations by hand on an exam, you can now confidently distinguish between their underlying logic.

Key Takeaways:

Generalized Fisher's Exact Test: The theoretical "gold standard." It extends the 2x2 method by calculating the exact probability of all possible tables. Its primary limitation is that it is computationally infeasible for most r x c tables.
Permutation Chi-Square Test: A practical, simulation-based alternative. It generates a null distribution by shuffling the data, and the p-value is the proportion of simulated $χ^{2}$ statistics that are more extreme than the observed $χ^{2}$ .
The Key Distinction: Fisher's is an exact method based on exhaustive enumeration and hypergeometric probabilities. The permutation test is an approximate method based on resampling and the $χ^{2}$ statistic.

In our next lesson, we will return to 2x2 tables to answer a different kind of question. Having established if an association exists, we will learn how to measure its strength and direction by deciding when to report the relative risk versus the odds ratio.