{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 36: Neyman's Repeated-Sampling Variance\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "The exact variance of the difference in means over all possible randomisations, why it cannot be estimated without bias, and why the standard estimator is conservative.\n", "\n", "**Learning outcomes**\n", "\n", "- State Neyman's finite-population variance formula and compute it from a science table.\n", "- Derive the variance of a treated-group mean under complete randomisation.\n", "- Prove that the Neyman variance estimator is conservative, and when it is unbiased.\n", "- Compute Neyman standard errors and confidence intervals for RCTs with continuous and binary outcomes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The finite-population variance\n", "\n", "Neyman (1923) treated potential outcomes as fixed and asked how the difference in means $\\hat\\tau = \\bar Y_1 - \\bar Y_0$ varies across all possible completely randomised assignments. With finite-population variances (divisor $n - 1$)\n", "\n", "$$\n", "S_1^2 = \\tfrac{1}{n-1}\\sum_i(Y_i(1) - \\bar Y(1))^2, \\quad S_0^2 = \\tfrac{1}{n-1}\\sum_i(Y_i(0) - \\bar Y(0))^2, \\quad S_\\tau^2 = \\tfrac{1}{n-1}\\sum_i(\\tau_i - \\tau_{fp})^2,\n", "$$\n", "\n", "the exact variance is\n", "\n", "$$\n", "\\operatorname{Var}(\\hat\\tau) = \\frac{S_1^2}{n_1} + \\frac{S_0^2}{n_0} - \\frac{S_\\tau^2}{n}.\n", "$$\n", "\n", "### Estimating it\n", "\n", "$S_1^2$ and $S_0^2$ are estimated without bias by the sample variances $s_1^2$ (treated) and $s_0^2$ (control). But $S_\\tau^2$ involves both potential outcomes of each unit and cannot be estimated. Neyman's estimator drops it:\n", "\n", "$$\n", "\\hat V = \\frac{s_1^2}{n_1} + \\frac{s_0^2}{n_0}, \\qquad E[\\hat V] - \\operatorname{Var}(\\hat\\tau) = \\frac{S_\\tau^2}{n} \\ge 0.\n", "$$\n", "\n", "So $\\hat V$ is **conservative**: unbiased when effects are constant ($S_\\tau^2 = 0$), too large otherwise. Confidence intervals $\\hat\\tau \\pm 1.96\\sqrt{\\hat V}$ therefore cover *at least* 95% in large samples.\n", "\n", "A sharper (still valid) bound uses $S_\\tau^2 \\ge (S_1 - S_0)^2$.\n", "\n", "### Finite versus super population\n", "\n", "If units are a random sample from a large population, the variance of $\\hat\\tau$ around the *super-population* ATE is $\\sigma_1^2/n_1 + \\sigma_0^2/n_0$, with no negative term, and $\\hat V$ is unbiased for it (Imbens & Rubin 2015). The same formula therefore serves both views: exact for one, conservative for the other.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Neyman's estimator\n", "\n", "A cash-transfer RCT has 100 treated and 100 control households, with sample sds of monthly consumption \\$40 and \\$30. Compute $\\hat V$ and the SE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\hat V = 40^2/100 + 30^2/100 = 16 + 9 = 25$.\n", "\n", "**Step 2.** SE $= \\sqrt{25} = 5$.\n", "\n", "**Answer:** \\$5.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Constant effects\n", "\n", "$S_1^2 = S_0^2 = 100$, $n_1 = n_0 = 50$, and the effect is the same for everyone. Compute $\\operatorname{Var}(\\hat\\tau)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Constant effects mean $S_\\tau^2 = 0$.\n", "\n", "**Step 2.** $\\operatorname{Var} = 100/50 + 100/50 - 0 = 4$.\n", "\n", "**Answer:** 4 (SE 2).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Heterogeneous effects\n", "\n", "Same as Exercise 2 but $S_\\tau^2 = 36$ ($n = 100$). Compute the true variance and the bias of $\\hat V$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{Var} = 2 + 2 - 36/100 = 3.64$.\n", "\n", "**Step 2.** $E[\\hat V] = 4$, so the bias is $0.36$ (about 10% of the variance; 5% of the SE).\n", "\n", "**Answer:** 3.64; Neyman's estimator overstates it by 0.36.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · A confidence interval\n", "\n", "The estimated effect in Exercise 1 is \\$12. Build a 95% CI.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $12 \\pm 1.96 \\times 5 = 12 \\pm 9.8$.\n", "\n", "**Answer:** $[2.2,\\; 21.8]$ dollars per month: positive, but imprecise.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Allocating a fixed sample\n", "\n", "With 200 households in total and outcome sds 40 (treated) and 30 (control), how should they be split to minimise $\\hat V$?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Minimise $S_1^2/n_1 + S_0^2/n_0$ with $n_1 + n_0 = 200$: the Lagrange solution (Day 26) is $n_1/n_0 = S_1/S_0 = 40/30$.\n", "\n", "**Step 2.** $n_1 = 200 \\times 40/70 \\approx 114$, $n_0 \\approx 86$.\n", "\n", "**Answer:** about 114 treated and 86 control.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Unbiased group means\n", "\n", "Under complete randomisation, prove $E[\\bar Y_1] = \\bar Y(1)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\bar Y_1 = \\frac{1}{n_1}\\sum_iT_iY_i(1)$.\n", "\n", "**Step 2.** $E[T_i] = n_1/n$ (Day 35, Exercise 7).\n", "\n", "**Step 3.** $E[\\bar Y_1] = \\frac{1}{n_1}\\cdot\\frac{n_1}{n}\\sum_iY_i(1) = \\bar Y(1)$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Variance of the treated mean\n", "\n", "Prove $\\operatorname{Var}(\\bar Y_1) = S_1^2\\big(\\frac{1}{n_1} - \\frac1n\\big)$.\n", "\n", "
Show solution\n", "\n", "Write $a_i = Y_i(1)$ and use Day 35's moments: $\\operatorname{Var}(T_i) = \\frac{n_1n_0}{n^2}$, $\\operatorname{Cov}(T_i, T_j) = -\\frac{n_1n_0}{n^2(n - 1)}$.\n", "\n", "**Step 1.** $\\operatorname{Var}\\Big(\\sum_iT_ia_i\\Big) = \\frac{n_1n_0}{n^2}\\Big[\\sum_ia_i^2 - \\frac{1}{n - 1}\\sum_{i \\ne j}a_ia_j\\Big]$.\n", "\n", "**Step 2.** Use $\\sum_{i \\ne j}a_ia_j = (\\sum_ia_i)^2 - \\sum_ia_i^2$. The bracket becomes\n", "\n", "$$\n", "\\frac{n\\sum_ia_i^2 - (\\sum_ia_i)^2}{n - 1} = \\frac{n\\sum_i(a_i - \\bar a)^2}{n - 1} = nS_1^2.\n", "$$\n", "\n", "**Step 3.** So $\\operatorname{Var}(\\sum_iT_ia_i) = \\frac{n_1n_0}{n}S_1^2$.\n", "\n", "**Step 4.** Divide by $n_1^2$: $\\operatorname{Var}(\\bar Y_1) = \\frac{n_0}{nn_1}S_1^2 = S_1^2\\Big(\\frac{1}{n_1} - \\frac1n\\Big)$. $\\blacksquare$\n", "\n", "The $-1/n$ is a finite-population correction: treating everyone ($n_1 = n$) gives zero variance.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · The full Neyman formula (sketch)\n", "\n", "Using Exercise 7 and $\\operatorname{Cov}(\\bar Y_1, \\bar Y_0) = -S_{10}/n$, where $S_{10}$ is the finite-population covariance of $Y(1)$ and $Y(0)$, derive $\\operatorname{Var}(\\hat\\tau)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{Var}(\\hat\\tau) = \\operatorname{Var}(\\bar Y_1) + \\operatorname{Var}(\\bar Y_0) - 2\\operatorname{Cov}(\\bar Y_1, \\bar Y_0)$.\n", "\n", "**Step 2.** Substitute: $S_1^2\\big(\\frac{1}{n_1} - \\frac1n\\big) + S_0^2\\big(\\frac{1}{n_0} - \\frac1n\\big) + \\frac{2S_{10}}{n}$.\n", "\n", "**Step 3.** Since $\\tau_i = Y_i(1) - Y_i(0)$, $S_\\tau^2 = S_1^2 + S_0^2 - 2S_{10}$.\n", "\n", "**Step 4.** So $-\\frac{S_1^2 + S_0^2 - 2S_{10}}{n} = -\\frac{S_\\tau^2}{n}$, giving $\\operatorname{Var}(\\hat\\tau) = \\frac{S_1^2}{n_1} + \\frac{S_0^2}{n_0} - \\frac{S_\\tau^2}{n}$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Neyman's estimator is conservative\n", "\n", "Given $E[s_1^2] = S_1^2$ and $E[s_0^2] = S_0^2$ under complete randomisation, prove $E[\\hat V] \\ge \\operatorname{Var}(\\hat\\tau)$, with equality iff effects are constant.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $E[\\hat V] = S_1^2/n_1 + S_0^2/n_0$.\n", "\n", "**Step 2.** Subtract the true variance: $E[\\hat V] - \\operatorname{Var}(\\hat\\tau) = S_\\tau^2/n \\ge 0$.\n", "\n", "**Step 3.** $S_\\tau^2 = 0$ iff all $\\tau_i$ equal $\\tau_{fp}$, i.e. constant effects. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · A sharper bound\n", "\n", "Prove $S_\\tau^2 \\ge (S_1 - S_0)^2$ and deduce a smaller conservative variance estimator.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $S_\\tau^2 = S_1^2 + S_0^2 - 2S_{10}$.\n", "\n", "**Step 2.** Cauchy–Schwarz (Day 16): $S_{10} \\le S_1S_0$.\n", "\n", "**Step 3.** So $S_\\tau^2 \\ge S_1^2 + S_0^2 - 2S_1S_0 = (S_1 - S_0)^2$.\n", "\n", "**Step 4.** Hence $\\operatorname{Var}(\\hat\\tau) \\le \\frac{S_1^2}{n_1} + \\frac{S_0^2}{n_0} - \\frac{(S_1 - S_0)^2}{n}$, estimated by plugging in $s_1, s_0$. $\\blacksquare$\n", "\n", "The bound is attained when $Y(1)$ and $Y(0)$ are perfectly positively correlated.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · School attendance\n", "\n", "A deworming RCT randomises 50 schools, 25 to each arm. Attendance rates (%) have sds 8 (treated) and 6 (control); the difference in means is 5 points. Compute the Neyman SE and a 95% CI.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\hat V = 8^2/25 + 6^2/25 = 2.56 + 1.44 = 4$.\n", "\n", "**Step 2.** SE $= 2$.\n", "\n", "**Step 3.** CI: $5 \\pm 1.96 \\times 2 = [1.08,\\; 8.92]$.\n", "\n", "**Answer:** attendance rose by 5 points (95% CI 1.1 to 8.9).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Using the sharper bound\n", "\n", "For Exercise 11, compute the variance bound of Exercise 10.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $(s_1 - s_0)^2/n = (8 - 6)^2/50 = 0.08$.\n", "\n", "**Step 2.** Bound $= 4 - 0.08 = 3.92$, SE $\\approx 1.98$.\n", "\n", "**Answer:** a slightly narrower but still valid interval; the gain is small unless the two sds differ a lot.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · The exact variance for the training table\n", "\n", "For Day 31's four workers ($Y(1) = (14, 12, 13, 15)$, $Y(0) = (10, 12, 8, 15)$, $n_1 = n_0 = 2$), compute $S_1^2$, $S_0^2$, $S_\\tau^2$ and Neyman's exact variance. Check it against the variance of the six estimates from Day 31, Exercise 11.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\bar Y(1) = 13.5$; deviations $0.5, -1.5, -0.5, 1.5$; $S_1^2 = (0.25 + 2.25 + 0.25 + 2.25)/3 = 5/3 \\approx 1.667$.\n", "\n", "**Step 2.** $\\bar Y(0) = 11.25$; squared deviations $1.5625, 0.5625, 10.5625, 14.0625$; $S_0^2 = 26.75/3 \\approx 8.917$.\n", "\n", "**Step 3.** $\\tau = (4, 0, 5, 0)$, mean 2.25; squared deviations $3.0625, 5.0625, 7.5625, 5.0625$; $S_\\tau^2 = 20.75/3 \\approx 6.917$.\n", "\n", "**Step 4.** $\\operatorname{Var} = 1.667/2 + 8.917/2 - 6.917/4 = 0.833 + 4.458 - 1.729 = 3.5625$.\n", "\n", "**Step 5.** The six estimates (1.5, 0, 4.5, 0, 4.5, 3) have mean 2.25 and variance $\\frac16\\sum(\\hat\\tau - 2.25)^2 = 3.5625$. ✓\n", "\n", "**Answer:** 3.5625 exactly, as Neyman's formula says.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · A binary outcome\n", "\n", "A smoking-cessation trial has 200 per arm; 30% quit with the intervention and 40% with usual care. Compute the difference, Neyman SE and z-statistic.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Difference $= 0.30 - 0.40 = -0.10$.\n", "\n", "**Step 2.** Binary outcomes: $s^2 \\approx p(1 - p)$. $\\hat V = 0.21/200 + 0.24/200 = 0.00225$.\n", "\n", "**Step 3.** SE $\\approx 0.0474$; $z = -0.10/0.0474 \\approx -2.11$.\n", "\n", "**Answer:** the intervention *lowered* quit rates by 10 points ($p \\approx 0.035$), a surprising result worth checking for implementation problems.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Which variance to report?\n", "\n", "A study randomises all 120 municipalities of a province (finite population, Day 31 Exercise 14). Is $\\hat V$ exact, conservative or anti-conservative, and how should results be described?\n", "\n", "
Show solution\n", "\n", "**Step 1.** For the finite-population ATE, $\\hat V$ overstates the variance by $S_\\tau^2/n$ on average: conservative.\n", "\n", "**Step 2.** If effects are roughly constant across municipalities, the overstatement is negligible.\n", "\n", "**Answer:** report $\\hat V$-based CIs as (slightly) conservative for the effect in *these* 120 municipalities; for other provinces, external validity applies.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import itertools\n", "import numpy as np\n", "\n", "y1 = np.array([14, 12, 13, 15.]); y0 = np.array([10, 12, 8, 15.]); n, n1 = 4, 2\n", "S1, S0, St = y1.var(ddof=1), y0.var(ddof=1), (y1 - y0).var(ddof=1)\n", "print(f\"S1^2 {S1:.3f}, S0^2 {S0:.3f}, Stau^2 {St:.3f}; Neyman variance {S1 / n1 + S0 / (n - n1) - St / n:.4f}\")\n", "ests = []\n", "for c in itertools.combinations(range(n), n1):\n", " T = np.zeros(n, bool); T[list(c)] = True\n", " ests.append(y1[T].mean() - y0[~T].mean())\n", "print(\"variance over all assignments:\", np.var(ests))\n", "\n", "# Coverage of Neyman intervals with heterogeneous effects\n", "rng = np.random.default_rng(36)\n", "N, N1 = 200, 100\n", "Y0 = rng.normal(50, 10, N); Y1 = Y0 + rng.normal(5, 6, N) # heterogeneous effects\n", "tau = (Y1 - Y0).mean(); cover = []; vhat = []\n", "for _ in range(20_000):\n", " T = np.zeros(N, bool); T[rng.choice(N, N1, replace=False)] = True\n", " est = Y1[T].mean() - Y0[~T].mean()\n", " v = Y1[T].var(ddof=1) / N1 + Y0[~T].var(ddof=1) / (N - N1)\n", " vhat.append(v); cover.append(abs(est - tau) <= 1.96 * np.sqrt(v))\n", "true_v = Y1.var(ddof=1) / N1 + Y0.var(ddof=1) / (N - N1) - (Y1 - Y0).var(ddof=1) / N\n", "print(f\"true variance {true_v:.3f}, mean Neyman estimate {np.mean(vhat):.3f}, coverage {np.mean(cover):.3f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Over all randomisations, $\\operatorname{Var}(\\hat\\tau) = S_1^2/n_1 + S_0^2/n_0 - S_\\tau^2/n$.\n", "- The last term needs both potential outcomes per unit, so it is dropped: Neyman's estimator is conservative.\n", "- It is exact under constant effects and unbiased for the super-population variance." ] } ] }