Day 37: Fisher's Exact Test & Randomisation Inference

Theory

The sharp null

Fisher (1935) tested the sharp null

H_0: Y_i(1) = Y_i(0) \quad\text{for every unit } i.

Under $H_0$ , each unit's outcome would have been the same whatever its assignment, so the observed $Y_i$ fills in both potential outcomes. The whole science table is known.

The randomisation distribution

Pick a test statistic $S(\mathbf T, \mathbf Y)$ , e.g. the difference in means. Because the science table is known under $H_0$ , we can compute $S$ for every assignment the design could have produced. The exact p-value is

p = \frac{\#\{\text{assignments } \mathbf t: |S(\mathbf t, \mathbf Y)| \ge |S^{obs}|\}}{\#\{\text{assignments}\}}.

No normality, no large sample, no model: the only randomness is the one the researcher created. When there are too many assignments to list, draw $R$ at random (Monte Carlo); the p-value then has standard error $\sqrt{p(1 - p)/R}$ .

Choices and extensions

Statistic: any function of the data. Rank statistics (Wilcoxon) resist outliers.
Other sharp nulls: $H_0: Y_i(1) = Y_i(0) + \tau_0$ for all $i$ . Subtract $\tau_0$ from treated outcomes and test again. Collecting all $\tau_0$ not rejected gives a confidence interval (test inversion).
Design: permute exactly as the design randomised: within blocks for blocked designs, whole clusters for cluster designs.
Sharp versus weak null: the sharp null (no effect for anyone) is stronger than Neyman's weak null (zero average effect). Randomisation tests are exact for the former and only approximately valid for the latter.

Part 1 · Warm-up

Exercise 1 · How many assignments?

A pilot randomises 4 of 8 villages to a microfinance programme. How many assignments are possible, and what is the smallest achievable two-sided p-value?

Show solution

Step 1. $\binom84 = 70$ assignments.

Step 2. The observed assignment always counts itself. With a symmetric two-sided statistic, its mirror image (treated and control swapped) gives the same $|S|$ , so at least 2 of 70 qualify.

Answer: 70 assignments; smallest two-sided p-value $2/70 \approx 0.029$ .

Exercise 2 · Imputing under the sharp null

A treated village has profit 12 and a control village 9. Under $H_0$ , what are their missing potential outcomes?

Show solution

Step 1. No effect for anyone: $Y(0) = Y(1)$ for each village.

Step 2. Treated village: $Y(0) = 12$ . Control village: $Y(1) = 9$ .

Exercise 3 · The statistic under another assignment

Profits are 12, 15, 11, 14 (treated villages A–D) and 9, 10, 13, 8 (controls E–H). Under $H_0$ , what difference in means would the assignment {A, B, E, F} have produced?

Show solution

Step 1. Under $H_0$ outcomes do not move: "treated" A, B, E, F have $(12 + 15 + 9 + 10)/4 = 11.5$ .

Step 2. "Controls" C, D, G, H have $(11 + 14 + 13 + 8)/4 = 11.5$ .

Answer: 0.

Exercise 4 · From counts to a p-value

Of the 70 assignments, 8 give $|\bar Y_1 - \bar Y_0| \ge 3$ (the observed value). What is the p-value?

Show solution

Step 1. $p = 8/70 \approx 0.114$ .

Answer: 0.11: not significant at 5% or 10%.

Exercise 5 · Monte Carlo precision

With 70 million possible assignments you sample $R = 1{,}000$ . If the true p-value is 0.05, what is the Monte Carlo SE? How many draws give an SE of 0.002?

Show solution

Step 1. SE $= \sqrt{0.05 \times 0.95/1000} \approx 0.0069$ .

Step 2. Solve $\sqrt{0.0475/R} = 0.002$ : $R = 0.0475/0.000004 \approx 11{,}875$ .

Answer: SE ≈ 0.007 with 1,000 draws; about 12,000 draws for SE 0.002.

Part 2 · Proofs

Exercise 6 · The test is exact

Under the sharp null and the actual design, prove that $P(p \le \alpha) \le \alpha$ .

Show solution

Step 1. Under $H_0$ the outcome vector $\mathbf Y$ is fixed whatever the assignment, so the statistic $S(\mathbf t)$ is a fixed number for each of the $K$ possible assignments.

Step 2. The observed assignment is one of them, chosen uniformly at random by the design. So $|S^{obs}|$ is a uniformly random draw from the $K$ values $|S(\mathbf t)|$ .

Step 3. The p-value is the share of values at least as large as the draw. A uniformly random draw lands in the top $100\alpha\%$ with probability at most $\alpha$ (ties only make the p-value larger).

Answer: $P(p \le \alpha) \le \alpha$ : exact Type I error control in any sample size. $\blacksquare$

Exercise 7 · Confidence intervals by inversion

For $H_0(\tau_0): Y_i(1) = Y_i(0) + \tau_0$ , show how to impute the science table, and explain why $\{\tau_0: p(\tau_0) > \alpha\}$ is a $1 - \alpha$ confidence set.

Show solution

Step 1. Treated units: $Y_i(0) = Y_i - \tau_0$ . Controls: $Y_i(1) = Y_i + \tau_0$ .

Step 2. Equivalently, subtract $\tau_0$ from treated outcomes; under $H_0(\tau_0)$ the adjusted outcomes satisfy the sharp null of no effect, so Exercise 6 applies.

Step 3. If the true effect is $\tau^*$ , the test of $H_0(\tau^*)$ rejects with probability at most $\alpha$ , so $\tau^*$ is in the set with probability at least $1 - \alpha$ . $\blacksquare$

Exercise 8 · The null distribution is centred at zero

Under the sharp null and complete randomisation, show the randomisation distribution of the difference in means has mean zero.

Show solution

Step 1. Under $H_0$ , $Y_i(1) = Y_i(0) = Y_i$ , so the finite-population ATE is 0.

Step 2. By Day 31, Exercise 8, the difference in means averages to the ATE over all assignments.

Answer: mean 0. $\blacksquare$

Exercise 9 · Sharp versus weak null

Construct a two-unit example where the weak null (ATE = 0) holds but the sharp null fails.

Show solution

Step 1. Unit 1: $Y(0) = 0$ , $Y(1) = 5$ (effect +5). Unit 2: $Y(0) = 5$ , $Y(1) = 0$ (effect −5).

Step 2. ATE $= (5 - 5)/2 = 0$ : the weak null holds.

Step 3. Individual effects are not zero: the sharp null fails.

Answer: a randomisation test of the sharp null can reject here even though the average effect is zero; it tests "no effect for anyone".

Exercise 10 · Rank statistics

Show that, under the sharp null, the randomisation distribution of the Wilcoxon rank-sum statistic (the sum of treated units' ranks) depends only on $n$ and $n_1$ (with no ties), not on the outcome values.

Show solution

Step 1. Under $H_0$ the outcomes, hence their ranks $1, \dots, n$ , are fixed.

Step 2. Each assignment picks $n_1$ of these ranks; the statistic is their sum.

Step 3. The set of possible sums and their frequencies depend only on which subsets of $\{1, \dots, n\}$ of size $n_1$ exist.

Answer: the null distribution can be tabulated once for each $(n, n_1)$ ; an extreme outlier changes one rank by at most a few places, so the test is robust.

Part 3 · Applied

Exercise 11 · A microfinance pilot

Using the eight villages of Exercise 3 (treated 12, 15, 11, 14; control 9, 10, 13, 8), compute the observed difference in means, the exact one-sided and two-sided p-values (see the code), and compare with a t-test.

Show solution

Step 1. Treated mean $= 52/4 = 13$ ; control mean $= 40/4 = 10$ ; observed difference 3.

Step 2. Enumerate all 70 assignments under $H_0$ : 4 give a difference $\ge 3$ and 8 give $|\text{difference}| \ge 3$ .

Step 3. One-sided $p = 4/70 \approx 0.057$ ; two-sided $p = 8/70 \approx 0.114$ .

Step 4. A Welch t-test gives $p \approx 0.08$ , somewhat smaller, but it relies on approximate normality with only 8 villages.

Answer: no significant evidence at 5%. The randomisation p-value is exact whatever the distribution of profits.

Exercise 12 · A small hospital trial

10 patients are randomised, 5 per arm; all 5 treated recover versus 1 of 5 controls. Compute Fisher's exact p-values.

Show solution

Step 1. Under $H_0$ , the 6 recoverers would recover whatever the arm. The number of recoverers landing in the treated arm is hypergeometric: choose 5 treated from 10 patients, 6 of whom recover.

Step 2. All 5 treated recover: $\binom65\binom40/\binom{10}{5} = 6/252 \approx 0.024$ . That is the one-sided p-value (no table is more extreme).

Step 3. The opposite extreme (only 1 treated recoverer) also has probability $\binom61\binom44/252 = 6/252$ .

Step 4. Two-sided $p = 12/252 \approx 0.048$ .

Answer: one-sided 0.024, two-sided 0.048: significant at 5% with just 10 patients, and no large-sample approximation needed.

Exercise 13 · Testing a specific effect

For the microfinance pilot, test the sharp null that the programme raises every village's profit by exactly 2.

Show solution

Step 1. Subtract 2 from treated profits: 10, 13, 9, 12; controls stay 9, 10, 13, 8.

Step 2. The adjusted difference is $11 - 10 = 1$ ; the randomisation test of no effect on the adjusted data gives $p \approx 0.66$ (code).

Step 3. Repeating for every $\tau_0$ and keeping those with $p > 0.10$ gives a 90% interval; here it spans roughly 0 to 6.

Answer: an effect of 2 is entirely consistent with the data; the interval is wide because there are only 8 villages.

Exercise 14 · Clustered randomisation inference

A study randomises 10 of 20 schools; outcomes are measured for 3,000 pupils. How should the randomisation test permute the data, and why?

Show solution

Step 1. The design assigned schools, so the null distribution must come from re-assigning whole schools: $\binom{20}{10} = 184{,}756$ possibilities.

Step 2. Permuting individual pupils would generate assignments the design could never produce (mixed schools) and understate variability, because pupils in a school share shocks.

Answer: permute at the school level, using a school-level or pupil-level statistic computed for each re-assignment.

Exercise 15 · One treated state

A tax reform is adopted by 1 of 20 states. You compute a statistic for each state as if it had been the one treated. What is the smallest possible p-value, and what does that imply?

Show solution

Step 1. If the reform were assigned at random to one state, there are 20 equally likely possibilities.

Step 2. The smallest one-sided p-value is $1/20 = 0.05$ (the treated state is the most extreme of all 20).

Answer: 0.05. With one treated unit, randomisation inference can never be very decisive; this placebo logic underlies inference for synthetic control (Day 86). (Here the "as if random" assumption must be argued, since states choose reforms.)

Verify in Python

import itertools
import numpy as np
from scipy import stats

y = np.array([12, 15, 11, 14, 9, 10, 13, 8.]); treated = np.array([1, 1, 1, 1, 0, 0, 0, 0], bool)

def rand_test(y, treated, tau0=0.0):
    z = y - tau0 * treated                                   # impute under H0(tau0)
    obs = z[treated].mean() - z[~treated].mean()
    dist = np.array([z[list(c)].mean() - np.delete(z, list(c)).mean()
                     for c in itertools.combinations(range(len(z)), treated.sum())])
    return obs, (dist >= obs - 1e-9).mean(), (np.abs(dist) >= abs(obs) - 1e-9).mean()

obs, p1, p2 = rand_test(y, treated)
print(f"observed {obs}, one-sided p {p1:.3f}, two-sided p {p2:.3f}")
print("Welch t-test p:", round(stats.ttest_ind(y[treated], y[~treated], equal_var=False).pvalue, 3))
print("test of tau0 = 2: p =", round(rand_test(y, treated, 2)[2], 3))
ci = [t for t in np.arange(-4, 10, 0.1) if rand_test(y, treated, t)[2] > 0.10]
print(f"90% interval by inversion: [{min(ci):.1f}, {max(ci):.1f}]")

# Fisher's exact test for the hospital trial
print("Fisher exact (two-sided):", round(stats.fisher_exact([[5, 0], [1, 4]])[1], 4),
      " one-sided:", round(stats.fisher_exact([[5, 0], [1, 4]], alternative="greater")[1], 4))

Key takeaways

Under the sharp null, every potential outcome is known, so the randomisation itself gives an exact null distribution.
Permute exactly as the design randomised (within blocks, by cluster).
Inverting sharp-null tests yields confidence intervals; randomisation tests concern "no effect for anyone".

Day 37. Fisher's Exact Test & Randomisation Inference

Theory

The sharp null

The randomisation distribution

Choices and extensions

Part 1 · Warm-up

Exercise 1 · How many assignments?

Exercise 2 · Imputing under the sharp null

Exercise 3 · The statistic under another assignment

Exercise 4 · From counts to a p-value

Exercise 5 · Monte Carlo precision

Part 2 · Proofs

Exercise 6 · The test is exact

Exercise 7 · Confidence intervals by inversion

Exercise 8 · The null distribution is centred at zero

Exercise 9 · Sharp versus weak null

Exercise 10 · Rank statistics

Part 3 · Applied

Exercise 11 · A microfinance pilot

Exercise 12 · A small hospital trial

Exercise 13 · Testing a specific effect

Exercise 14 · Clustered randomisation inference

Exercise 15 · One treated state

Verify in Python

Key takeaways

Simulation Lab: Fisher's Exact Randomisation Test