{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 37: Fisher's Exact Test & Randomisation Inference\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Exact p-values from the randomisation itself, under the sharp null of no effect for anyone, with no distributional assumptions.\n", "\n", "**Learning outcomes**\n", "\n", "- State the sharp null hypothesis and impute the full science table under it.\n", "- Compute an exact randomisation p-value by enumerating or simulating assignments.\n", "- Prove the validity of the randomisation test and invert tests to get confidence intervals.\n", "- Apply randomisation inference to small trials, binary outcomes, clustered designs and single treated units." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The sharp null\n", "\n", "Fisher (1935) tested the **sharp null**\n", "\n", "$$\n", "H_0: Y_i(1) = Y_i(0) \\quad\\text{for every unit } i.\n", "$$\n", "\n", "Under $H_0$, each unit's outcome would have been the same whatever its assignment, so the observed $Y_i$ fills in *both* potential outcomes. The whole science table is known.\n", "\n", "### The randomisation distribution\n", "\n", "Pick a test statistic $S(\\mathbf T, \\mathbf Y)$, e.g. the difference in means. Because the science table is known under $H_0$, we can compute $S$ for *every* assignment the design could have produced. The exact p-value is\n", "\n", "$$\n", "p = \\frac{\\#\\{\\text{assignments } \\mathbf t: |S(\\mathbf t, \\mathbf Y)| \\ge |S^{obs}|\\}}{\\#\\{\\text{assignments}\\}}.\n", "$$\n", "\n", "No normality, no large sample, no model: the only randomness is the one the researcher created. When there are too many assignments to list, draw $R$ at random (Monte Carlo); the p-value then has standard error $\\sqrt{p(1 - p)/R}$.\n", "\n", "### Choices and extensions\n", "\n", "- **Statistic:** any function of the data. Rank statistics (Wilcoxon) resist outliers.\n", "- **Other sharp nulls:** $H_0: Y_i(1) = Y_i(0) + \\tau_0$ for all $i$. Subtract $\\tau_0$ from treated outcomes and test again. Collecting all $\\tau_0$ not rejected gives a **confidence interval** (test inversion).\n", "- **Design:** permute exactly as the design randomised: within blocks for blocked designs, whole clusters for cluster designs.\n", "- **Sharp versus weak null:** the sharp null (no effect for anyone) is stronger than Neyman's weak null (zero *average* effect). Randomisation tests are exact for the former and only approximately valid for the latter.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · How many assignments?\n", "\n", "A pilot randomises 4 of 8 villages to a microfinance programme. How many assignments are possible, and what is the smallest achievable two-sided p-value?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\binom84 = 70$ assignments.\n", "\n", "**Step 2.** The observed assignment always counts itself. With a symmetric two-sided statistic, its mirror image (treated and control swapped) gives the same $|S|$, so at least 2 of 70 qualify.\n", "\n", "**Answer:** 70 assignments; smallest two-sided p-value $2/70 \\approx 0.029$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Imputing under the sharp null\n", "\n", "A treated village has profit 12 and a control village 9. Under $H_0$, what are their missing potential outcomes?\n", "\n", "
Show solution\n", "\n", "**Step 1.** No effect for anyone: $Y(0) = Y(1)$ for each village.\n", "\n", "**Step 2.** Treated village: $Y(0) = 12$. Control village: $Y(1) = 9$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · The statistic under another assignment\n", "\n", "Profits are 12, 15, 11, 14 (treated villages A–D) and 9, 10, 13, 8 (controls E–H). Under $H_0$, what difference in means would the assignment {A, B, E, F} have produced?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Under $H_0$ outcomes do not move: \"treated\" A, B, E, F have $(12 + 15 + 9 + 10)/4 = 11.5$.\n", "\n", "**Step 2.** \"Controls\" C, D, G, H have $(11 + 14 + 13 + 8)/4 = 11.5$.\n", "\n", "**Answer:** 0.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · From counts to a p-value\n", "\n", "Of the 70 assignments, 8 give $|\\bar Y_1 - \\bar Y_0| \\ge 3$ (the observed value). What is the p-value?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $p = 8/70 \\approx 0.114$.\n", "\n", "**Answer:** 0.11: not significant at 5% or 10%.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Monte Carlo precision\n", "\n", "With 70 million possible assignments you sample $R = 1{,}000$. If the true p-value is 0.05, what is the Monte Carlo SE? How many draws give an SE of 0.002?\n", "\n", "
Show solution\n", "\n", "**Step 1.** SE $= \\sqrt{0.05 \\times 0.95/1000} \\approx 0.0069$.\n", "\n", "**Step 2.** Solve $\\sqrt{0.0475/R} = 0.002$: $R = 0.0475/0.000004 \\approx 11{,}875$.\n", "\n", "**Answer:** SE ≈ 0.007 with 1,000 draws; about 12,000 draws for SE 0.002.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · The test is exact\n", "\n", "Under the sharp null and the actual design, prove that $P(p \\le \\alpha) \\le \\alpha$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Under $H_0$ the outcome vector $\\mathbf Y$ is fixed whatever the assignment, so the statistic $S(\\mathbf t)$ is a fixed number for each of the $K$ possible assignments.\n", "\n", "**Step 2.** The observed assignment is one of them, chosen uniformly at random by the design. So $|S^{obs}|$ is a uniformly random draw from the $K$ values $|S(\\mathbf t)|$.\n", "\n", "**Step 3.** The p-value is the share of values at least as large as the draw. A uniformly random draw lands in the top $100\\alpha\\%$ with probability at most $\\alpha$ (ties only make the p-value larger).\n", "\n", "**Answer:** $P(p \\le \\alpha) \\le \\alpha$: exact Type I error control in any sample size. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Confidence intervals by inversion\n", "\n", "For $H_0(\\tau_0): Y_i(1) = Y_i(0) + \\tau_0$, show how to impute the science table, and explain why $\\{\\tau_0: p(\\tau_0) > \\alpha\\}$ is a $1 - \\alpha$ confidence set.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Treated units: $Y_i(0) = Y_i - \\tau_0$. Controls: $Y_i(1) = Y_i + \\tau_0$.\n", "\n", "**Step 2.** Equivalently, subtract $\\tau_0$ from treated outcomes; under $H_0(\\tau_0)$ the adjusted outcomes satisfy the sharp null of no effect, so Exercise 6 applies.\n", "\n", "**Step 3.** If the true effect is $\\tau^*$, the test of $H_0(\\tau^*)$ rejects with probability at most $\\alpha$, so $\\tau^*$ is in the set with probability at least $1 - \\alpha$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · The null distribution is centred at zero\n", "\n", "Under the sharp null and complete randomisation, show the randomisation distribution of the difference in means has mean zero.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Under $H_0$, $Y_i(1) = Y_i(0) = Y_i$, so the finite-population ATE is 0.\n", "\n", "**Step 2.** By Day 31, Exercise 8, the difference in means averages to the ATE over all assignments.\n", "\n", "**Answer:** mean 0. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Sharp versus weak null\n", "\n", "Construct a two-unit example where the weak null (ATE = 0) holds but the sharp null fails.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Unit 1: $Y(0) = 0$, $Y(1) = 5$ (effect +5). Unit 2: $Y(0) = 5$, $Y(1) = 0$ (effect −5).\n", "\n", "**Step 2.** ATE $= (5 - 5)/2 = 0$: the weak null holds.\n", "\n", "**Step 3.** Individual effects are not zero: the sharp null fails.\n", "\n", "**Answer:** a randomisation test of the sharp null can reject here even though the average effect is zero; it tests \"no effect for anyone\".\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Rank statistics\n", "\n", "Show that, under the sharp null, the randomisation distribution of the Wilcoxon rank-sum statistic (the sum of treated units' ranks) depends only on $n$ and $n_1$ (with no ties), not on the outcome values.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Under $H_0$ the outcomes, hence their ranks $1, \\dots, n$, are fixed.\n", "\n", "**Step 2.** Each assignment picks $n_1$ of these ranks; the statistic is their sum.\n", "\n", "**Step 3.** The set of possible sums and their frequencies depend only on which subsets of $\\{1, \\dots, n\\}$ of size $n_1$ exist.\n", "\n", "**Answer:** the null distribution can be tabulated once for each $(n, n_1)$; an extreme outlier changes one rank by at most a few places, so the test is robust.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · A microfinance pilot\n", "\n", "Using the eight villages of Exercise 3 (treated 12, 15, 11, 14; control 9, 10, 13, 8), compute the observed difference in means, the exact one-sided and two-sided p-values (see the code), and compare with a t-test.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Treated mean $= 52/4 = 13$; control mean $= 40/4 = 10$; observed difference 3.\n", "\n", "**Step 2.** Enumerate all 70 assignments under $H_0$: 4 give a difference $\\ge 3$ and 8 give $|\\text{difference}| \\ge 3$.\n", "\n", "**Step 3.** One-sided $p = 4/70 \\approx 0.057$; two-sided $p = 8/70 \\approx 0.114$.\n", "\n", "**Step 4.** A Welch t-test gives $p \\approx 0.08$, somewhat smaller, but it relies on approximate normality with only 8 villages.\n", "\n", "**Answer:** no significant evidence at 5%. The randomisation p-value is exact whatever the distribution of profits.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · A small hospital trial\n", "\n", "10 patients are randomised, 5 per arm; all 5 treated recover versus 1 of 5 controls. Compute Fisher's exact p-values.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Under $H_0$, the 6 recoverers would recover whatever the arm. The number of recoverers landing in the treated arm is hypergeometric: choose 5 treated from 10 patients, 6 of whom recover.\n", "\n", "**Step 2.** All 5 treated recover: $\\binom65\\binom40/\\binom{10}{5} = 6/252 \\approx 0.024$. That is the one-sided p-value (no table is more extreme).\n", "\n", "**Step 3.** The opposite extreme (only 1 treated recoverer) also has probability $\\binom61\\binom44/252 = 6/252$.\n", "\n", "**Step 4.** Two-sided $p = 12/252 \\approx 0.048$.\n", "\n", "**Answer:** one-sided 0.024, two-sided 0.048: significant at 5% with just 10 patients, and no large-sample approximation needed.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Testing a specific effect\n", "\n", "For the microfinance pilot, test the sharp null that the programme raises *every* village's profit by exactly 2.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Subtract 2 from treated profits: 10, 13, 9, 12; controls stay 9, 10, 13, 8.\n", "\n", "**Step 2.** The adjusted difference is $11 - 10 = 1$; the randomisation test of no effect on the adjusted data gives $p \\approx 0.66$ (code).\n", "\n", "**Step 3.** Repeating for every $\\tau_0$ and keeping those with $p > 0.10$ gives a 90% interval; here it spans roughly 0 to 6.\n", "\n", "**Answer:** an effect of 2 is entirely consistent with the data; the interval is wide because there are only 8 villages.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Clustered randomisation inference\n", "\n", "A study randomises 10 of 20 schools; outcomes are measured for 3,000 pupils. How should the randomisation test permute the data, and why?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The design assigned *schools*, so the null distribution must come from re-assigning whole schools: $\\binom{20}{10} = 184{,}756$ possibilities.\n", "\n", "**Step 2.** Permuting individual pupils would generate assignments the design could never produce (mixed schools) and understate variability, because pupils in a school share shocks.\n", "\n", "**Answer:** permute at the school level, using a school-level or pupil-level statistic computed for each re-assignment.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · One treated state\n", "\n", "A tax reform is adopted by 1 of 20 states. You compute a statistic for each state as if it had been the one treated. What is the smallest possible p-value, and what does that imply?\n", "\n", "
Show solution\n", "\n", "**Step 1.** If the reform were assigned at random to one state, there are 20 equally likely possibilities.\n", "\n", "**Step 2.** The smallest one-sided p-value is $1/20 = 0.05$ (the treated state is the most extreme of all 20).\n", "\n", "**Answer:** 0.05. With one treated unit, randomisation inference can never be very decisive; this placebo logic underlies inference for synthetic control (Day 86). (Here the \"as if random\" assumption must be argued, since states choose reforms.)\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import itertools\n", "import numpy as np\n", "from scipy import stats\n", "\n", "y = np.array([12, 15, 11, 14, 9, 10, 13, 8.]); treated = np.array([1, 1, 1, 1, 0, 0, 0, 0], bool)\n", "\n", "def rand_test(y, treated, tau0=0.0):\n", " z = y - tau0 * treated # impute under H0(tau0)\n", " obs = z[treated].mean() - z[~treated].mean()\n", " dist = np.array([z[list(c)].mean() - np.delete(z, list(c)).mean()\n", " for c in itertools.combinations(range(len(z)), treated.sum())])\n", " return obs, (dist >= obs - 1e-9).mean(), (np.abs(dist) >= abs(obs) - 1e-9).mean()\n", "\n", "obs, p1, p2 = rand_test(y, treated)\n", "print(f\"observed {obs}, one-sided p {p1:.3f}, two-sided p {p2:.3f}\")\n", "print(\"Welch t-test p:\", round(stats.ttest_ind(y[treated], y[~treated], equal_var=False).pvalue, 3))\n", "print(\"test of tau0 = 2: p =\", round(rand_test(y, treated, 2)[2], 3))\n", "ci = [t for t in np.arange(-4, 10, 0.1) if rand_test(y, treated, t)[2] > 0.10]\n", "print(f\"90% interval by inversion: [{min(ci):.1f}, {max(ci):.1f}]\")\n", "\n", "# Fisher's exact test for the hospital trial\n", "print(\"Fisher exact (two-sided):\", round(stats.fisher_exact([[5, 0], [1, 4]])[1], 4),\n", " \" one-sided:\", round(stats.fisher_exact([[5, 0], [1, 4]], alternative=\"greater\")[1], 4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Under the sharp null, every potential outcome is known, so the randomisation itself gives an exact null distribution.\n", "- Permute exactly as the design randomised (within blocks, by cluster).\n", "- Inverting sharp-null tests yields confidence intervals; randomisation tests concern \"no effect for anyone\"." ] } ] }