{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 35: Randomisation & Identification\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "How random assignment makes treatment independent of everything that came before, what balance tables can and cannot tell you, and internal versus external validity.\n", "\n", "**Learning outcomes**\n", "\n", "- Describe Bernoulli, complete, stratified and cluster randomisation and compute their basic probabilities.\n", "- Prove that random assignment identifies the ATE and balances covariates in expectation.\n", "- Interpret balance tests and chance imbalance correctly.\n", "- Distinguish internal validity (randomisation) from external validity (random sampling)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Assignment mechanisms\n", "\n", "| Design | Rule | Number treated |\n", "|---|---|---|\n", "| Bernoulli | each unit independently with probability $p$ | random |\n", "| Complete | exactly $n_1$ of $n$, all $\\binom{n}{n_1}$ sets equally likely | fixed |\n", "| Stratified (blocked) | complete randomisation within blocks | fixed per block (Day 38) |\n", "| Cluster | whole groups assigned together | Day 40 |\n", "\n", "All share three properties (Imbens & Rubin 2015): **individualistic** (probabilities do not depend on other units' outcomes), **probabilistic** ($0 < P(T_i = 1) < 1$) and **unconfounded** (assignment does not depend on potential outcomes).\n", "\n", "### Why randomisation identifies the ATE\n", "\n", "Random assignment makes $T$ independent of $(Y(0), Y(1), X)$, all of which are fixed before assignment. Hence\n", "\n", "$$\n", "E[Y \\mid T = 1] - E[Y \\mid T = 0] = E[Y(1)] - E[Y(0)] = \\text{ATE}.\n", "$$\n", "\n", "Selection bias and heterogeneity bias (Day 34) are zero *by design*. This is identification without any model of outcomes.\n", "\n", "### Balance\n", "\n", "Independence implies **balance in expectation**: the treated and control groups have the same covariate distribution on average across randomisations. In any one experiment, chance imbalances occur. A balance table is a check that randomisation was implemented, not a test of whether it \"worked\": with 20 covariates, about one will differ \"significantly\" at 5% by chance alone. Large chance imbalance in a strong predictor of the outcome is best handled by design (blocking, Day 38) or pre-specified adjustment (Day 39).\n", "\n", "### Internal and external validity\n", "\n", "- **Internal validity:** the estimate is causal *for the units studied*. Random assignment delivers it.\n", "- **External validity:** the estimate applies to other populations, places or scales. Random *sampling* from the target population helps; random *assignment* does not.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Bernoulli surprises\n", "\n", "Ten clinics are each assigned to a new protocol by a fair coin flip. What is the probability that none is treated?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Each clinic is untreated with probability 0.5, independently.\n", "\n", "**Step 2.** $0.5^{10} = 1/1024 \\approx 0.001$.\n", "\n", "**Answer:** about 0.1%; small, but Bernoulli designs can produce lopsided splits, which complete randomisation rules out.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Counting complete randomisations\n", "\n", "A study randomises 10 of 20 schools to a reading programme. How many assignments are possible?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\binom{20}{10} = \\frac{20!}{10!\\,10!} = 184{,}756$.\n", "\n", "**Answer:** 184,756 equally likely assignments.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · How many get treated?\n", "\n", "200 farmers are each offered a subsidy with probability 0.3 (Bernoulli). Find the mean and sd of the number offered.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Number offered $\\sim$ Binomial(200, 0.3): mean $200 \\times 0.3 = 60$.\n", "\n", "**Step 2.** sd $= \\sqrt{200 \\times 0.3 \\times 0.7} = \\sqrt{42} \\approx 6.5$.\n", "\n", "**Answer:** about 60 ± 6.5.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Balance tests by chance\n", "\n", "A balance table compares treatment and control on 20 independent baseline covariates at the 5% level. How many \"significant\" differences should you expect, and what is the chance of at least one?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Expected number: $20 \\times 0.05 = 1$.\n", "\n", "**Step 2.** At least one: $1 - 0.95^{20} \\approx 0.64$.\n", "\n", "**Answer:** one, and a 64% chance of at least one, even with perfect randomisation.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Lotteries\n", "\n", "300 families apply for 100 places in an oversubscribed charter school, allocated by lottery. What is each applicant's probability of admission, and who is the study population?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $P(\\text{win}) = 100/300 = 1/3$ for every applicant.\n", "\n", "**Step 2.** The comparison is winners versus losers *among applicants*.\n", "\n", "**Answer:** 1/3. The lottery identifies the effect for families who apply, who may differ from non-applicants (external validity).\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Randomisation identifies the ATE\n", "\n", "Prove that if $T$ is independent of $(Y(0), Y(1))$ and $0 < P(T = 1) < 1$, then $E[Y \\mid T = 1] - E[Y \\mid T = 0] = \\text{ATE}$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $E[Y \\mid T = 1] = E[Y(1) \\mid T = 1]$ (observed outcome of the treated).\n", "\n", "**Step 2.** Independence: $E[Y(1) \\mid T = 1] = E[Y(1)]$.\n", "\n", "**Step 3.** Likewise $E[Y \\mid T = 0] = E[Y(0) \\mid T = 0] = E[Y(0)]$. ($0 < P(T = 1) < 1$ makes both conditional means defined.)\n", "\n", "**Step 4.** Difference: $E[Y(1)] - E[Y(0)] = \\text{ATE}$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Dependence in complete randomisation\n", "\n", "With $n_1$ of $n$ treated completely at random, find $P(T_i = 1)$, $P(T_i = 1, T_j = 1)$ for $i \\ne j$, and $\\operatorname{Cov}(T_i, T_j)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** By symmetry each unit is equally likely to be among the $n_1$: $P(T_i = 1) = n_1/n$.\n", "\n", "**Step 2.** Given $i$ is treated, $n_1 - 1$ of the remaining $n - 1$ slots are treated: $P(T_i = 1, T_j = 1) = \\frac{n_1}{n}\\cdot\\frac{n_1 - 1}{n - 1}$.\n", "\n", "**Step 3.** $\\operatorname{Cov} = \\frac{n_1(n_1 - 1)}{n(n - 1)} - \\frac{n_1^2}{n^2} = \\frac{n_1}{n}\\Big(\\frac{n_1 - 1}{n - 1} - \\frac{n_1}{n}\\Big) = -\\frac{n_1n_0}{n^2(n - 1)}$.\n", "\n", "**Answer:** slightly negative covariance (Day 1, Exercise 15); it drives the finite-population variance of Day 36.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Balance in expectation\n", "\n", "Under complete randomisation, prove $E[\\bar X_1 - \\bar X_0] = 0$ for any fixed covariate $X$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\bar X_1 = \\frac{1}{n_1}\\sum_iT_iX_i$, so $E[\\bar X_1] = \\frac{1}{n_1}\\sum_i\\frac{n_1}{n}X_i = \\bar X$.\n", "\n", "**Step 2.** Likewise $E[\\bar X_0] = \\bar X$.\n", "\n", "**Step 3.** Difference: 0. $\\blacksquare$\n", "\n", "Covariates behave like potential outcomes that treatment cannot change (Day 31, Exercise 8 with $Y(1) = Y(0) = X$).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Unbiased on average, not in every draw\n", "\n", "Suppose $Y_i(0) = X_i + \\varepsilon_i$ and the effect is constant, $\\tau$. Show that, *conditional on the realised assignment*, $\\bar Y_1 - \\bar Y_0 = \\tau + (\\bar X_1 - \\bar X_0) + (\\bar\\varepsilon_1 - \\bar\\varepsilon_0)$. What does this imply when you observe a large baseline imbalance?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Treated outcomes: $Y_i = X_i + \\varepsilon_i + \\tau$; control outcomes: $Y_i = X_i + \\varepsilon_i$.\n", "\n", "**Step 2.** Average within each group and subtract: $\\tau + (\\bar X_1 - \\bar X_0) + (\\bar\\varepsilon_1 - \\bar\\varepsilon_0)$.\n", "\n", "**Step 3.** Across randomisations the imbalance terms average to zero (Exercise 8), so the estimator is unbiased.\n", "\n", "**Step 4.** But in *this* experiment, if $\\bar X_1 - \\bar X_0$ is large, the estimate is shifted by it.\n", "\n", "**Answer:** a large observed imbalance in a predictive covariate tells you this particular estimate is likely off; adjusting for $X$ (Day 39) or blocking on it in advance (Day 38) fixes this.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Unequal probabilities\n", "\n", "Pupils in poor districts are treated with probability 0.8 and others with 0.2. Explain why a simple difference in means is biased for the ATE and how weighting by $1/P(T_i = t)$ fixes it.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The treated group over-represents poor districts (80% vs 20% treated), so if $Y(0)$ differs by district, the comparison mixes in district differences: selection bias by design.\n", "\n", "**Step 2.** Within each district, assignment is random, so district-specific differences are unbiased.\n", "\n", "**Step 3.** Weighting each unit by the inverse of its probability of the treatment it got re-creates the population in both arms: $E[TY/e] = E[Y(1)]$ and $E[(1 - T)Y/(1 - e)] = E[Y(0)]$ (Day 14).\n", "\n", "**Answer:** use IPW (or estimate by district and average with population shares). Known probabilities make this exact; Day 44 handles estimated ones.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · A charter-school lottery\n", "\n", "Among applicants, lottery winners score 0.25 sd higher in maths than losers two years later (SE 0.08). All winners enrol and no losers do. Interpret the estimate and its CI.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The lottery makes winning independent of potential scores, so 0.25 is an unbiased estimate of the ATE of charter attendance *for applicants*.\n", "\n", "**Step 2.** 95% CI: $0.25 \\pm 1.96 \\times 0.08 = [0.09,\\; 0.41]$.\n", "\n", "**Answer:** attending raised maths scores by about a quarter of a standard deviation (plausibly 0.1–0.4) for families who applied; whether it would do so for non-applicants is an external-validity question.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · A vaccine trial\n", "\n", "30,000 adults are randomised 2:1 to vaccine or placebo. There are 11 infections among 20,000 vaccinated and 185 among 10,000 on placebo. Compute the infection risks, the risk difference and vaccine efficacy.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Risks: vaccine $11/20{,}000 = 0.00055$; placebo $185/10{,}000 = 0.0185$.\n", "\n", "**Step 2.** Risk difference $= 0.00055 - 0.0185 \\approx -0.018$ (18 fewer infections per 1,000).\n", "\n", "**Step 3.** Efficacy $= 1 - 0.00055/0.0185 \\approx 1 - 0.030 = 0.97$.\n", "\n", "**Answer:** 97% efficacy. Randomisation means the arms differ only by vaccine (and chance), so the comparison is causal.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Microcredit in one city\n", "\n", "An RCT finds that offering microloans in one large city raised business investment by 20%. A ministry plans a national roll-out to rural areas. What does randomisation guarantee, and what does it not?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Internal validity: within the study city, the 20% is causal for the households studied.\n", "\n", "**Step 2.** External validity is not guaranteed: rural households differ in business opportunities, market access and existing credit.\n", "\n", "**Step 3.** Scale effects matter too: a national programme can change interest rates or local demand (SUTVA, Day 32).\n", "\n", "**Answer:** randomisation guarantees the first; the roll-out needs evidence from more sites, a model of heterogeneity (CATEs, Day 33) or a pilot in rural areas.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · An unlucky balance table\n", "\n", "In a job-training RCT, treated workers earned \\$500 more at baseline than controls (SE \\$400). The follow-up difference in earnings is \\$1,200. What should the analyst do?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The baseline difference has $z = 500/400 = 1.25$: consistent with chance, so no evidence that randomisation failed.\n", "\n", "**Step 2.** But baseline earnings strongly predict follow-up earnings, so part of the \\$1,200 may reflect the imbalance (Exercise 9).\n", "\n", "**Step 3.** Report the pre-specified covariate-adjusted estimate (Day 39) alongside the raw difference; do not drop or re-weight units ad hoc.\n", "\n", "**Answer:** keep the experiment, adjust for baseline earnings, and report both estimates.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Randomisation as the basis for inference\n", "\n", "Fisher argued that randomisation provides \"the reasoned basis\" for inference. In a study of 8 villages with 4 treated, how many assignments could have happened, and what probability statement does this enable without any model of outcomes?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\binom84 = 70$ equally likely assignments.\n", "\n", "**Step 2.** Under the sharp null of no effect for any village, every outcome is known whatever the assignment, so we can compute the test statistic for all 70.\n", "\n", "**Step 3.** The p-value is the share of the 70 assignments giving a statistic at least as extreme as the one observed.\n", "\n", "**Answer:** 70 assignments; randomisation alone yields an exact p-value (Day 37).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from math import comb\n", "\n", "print(\"assignments C(20,10):\", comb(20, 10))\n", "print(\"P(none treated) with 10 coin flips:\", 0.5**10)\n", "\n", "# Balance in expectation and chance imbalance across randomisations\n", "rng = np.random.default_rng(35)\n", "n, n1 = 100, 50\n", "X = rng.lognormal(10, 0.5, n) # baseline earnings, fixed\n", "y0 = X / 1000 + rng.normal(0, 3, n); y1 = y0 + 2 # constant effect of 2\n", "diffs, ests = [], []\n", "for _ in range(20_000):\n", " T = np.zeros(n, bool); T[rng.choice(n, n1, replace=False)] = True\n", " diffs.append(X[T].mean() - X[~T].mean())\n", " ests.append(y1[T].mean() - y0[~T].mean())\n", "diffs, ests = np.array(diffs), np.array(ests)\n", "print(f\"baseline imbalance: mean {diffs.mean():.0f}, sd {diffs.std():.0f} (mean ~0 relative to sd); mean estimate {ests.mean():.3f} (truth 2)\")\n", "print(\"corr(imbalance, estimate):\", round(np.corrcoef(diffs, ests)[0, 1], 2))\n", "\n", "# Vaccine trial (Exercise 12)\n", "r1, r0 = 11 / 20_000, 185 / 10_000\n", "print(f\"risks {r1:.5f} vs {r0:.4f}; efficacy {1 - r1 / r0:.3f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Random assignment makes treatment independent of potential outcomes and covariates, so the difference in means identifies the ATE.\n", "- Balance holds in expectation; chance imbalance in any one experiment is handled by blocking or pre-specified adjustment.\n", "- Randomisation buys internal validity; external validity needs more." ] } ] }