{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 40: Cluster Randomisation & Design Effects\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Randomising schools, clinics or villages instead of individuals; the intra-cluster correlation, the design effect, and why the number of clusters is what buys power.\n", "\n", "**Learning outcomes**\n", "\n", "- Compute the design effect, effective sample size and variance of a cluster-randomised difference in means.\n", "- Derive the variance of a cluster mean from the intra-cluster correlation.\n", "- Calculate the minimum detectable effect and the number of clusters needed for a target power.\n", "- Choose between individual and cluster randomisation and handle unequal cluster sizes and few clusters." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### When to randomise clusters\n", "\n", "Randomise groups (schools, clinics, villages, markets) when the intervention is delivered to groups (teacher training, a clinic protocol), when spillovers within groups are likely (Day 32), or when individual randomisation is politically or logistically impossible.\n", "\n", "### The intra-cluster correlation\n", "\n", "Write outcomes as $Y_{ig} = \\mu + u_g + e_{ig}$, with cluster effects $u_g$ (variance $\\sigma_u^2$) and individual noise $e_{ig}$ (variance $\\sigma_e^2$). Total variance $\\sigma^2 = \\sigma_u^2 + \\sigma_e^2$ and the **intra-cluster correlation**\n", "\n", "$$\n", "\\rho = \\frac{\\sigma_u^2}{\\sigma_u^2 + \\sigma_e^2}\n", "$$\n", "\n", "is the correlation between two people in the same cluster. Typical values: 0.01–0.05 for many health outcomes, 0.1–0.3 for test scores within schools.\n", "\n", "### Variance and design effect\n", "\n", "With clusters of size $m$, each cluster mean has variance $\\sigma^2[\\rho + (1 - \\rho)/m]$. With $J_1$ treated and $J_0$ control clusters,\n", "\n", "$$\n", "\\operatorname{Var}(\\hat\\tau) = \\sigma^2\\Big[\\rho + \\frac{1 - \\rho}{m}\\Big]\\Big(\\frac{1}{J_1} + \\frac{1}{J_0}\\Big) = \\underbrace{[1 + (m - 1)\\rho]}_{\\text{design effect}}\\times\\sigma^2\\Big(\\frac{1}{mJ_1} + \\frac{1}{mJ_0}\\Big).\n", "$$\n", "\n", "The **design effect** (DEFF) inflates the individually randomised variance. As $m \\to \\infty$ the variance falls only to $\\sigma^2\\rho(1/J_1 + 1/J_0)$: **adding clusters helps far more than adding people per cluster**.\n", "\n", "### Power\n", "\n", "With a share $P$ of $J$ clusters treated,\n", "\n", "$$\n", "\\text{MDE} = (z_{1-\\alpha/2} + z_{1-\\beta})\\sqrt{\\frac{\\sigma^2[\\rho + (1 - \\rho)/m]}{P(1 - P)J}}.\n", "$$\n", "\n", "Baseline covariates measured at the cluster level reduce the between-cluster variance and are especially valuable.\n", "\n", "### Analysis\n", "\n", "Use cluster-robust SEs (Day 29), or analyse cluster means; with few clusters, use randomisation inference at the cluster level (Day 37) or the wild cluster bootstrap. With unequal cluster sizes, decide whether the estimand weights individuals or clusters equally.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · A design effect\n", "\n", "A school-based trial has 30 pupils per school and an ICC of 0.10. Compute the design effect.\n", "\n", "
Show solution\n", "\n", "**Step 1.** DEFF $= 1 + (m - 1)\\rho = 1 + 29 \\times 0.10 = 3.9$.\n", "\n", "**Answer:** the variance is 3.9 times that of an individually randomised trial of the same size.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Effective sample size\n", "\n", "With 60 schools of 30 pupils (Exercise 1), what is the effective sample size?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Total pupils: $60 \\times 30 = 1{,}800$.\n", "\n", "**Step 2.** Effective $n = 1{,}800/3.9 \\approx 462$.\n", "\n", "**Answer:** 1,800 pupils carry the information of about 462 independent ones.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · SE and MDE\n", "\n", "Outcomes are standardised ($\\sigma = 1$), $\\rho = 0.1$, $m = 30$, and 30 schools per arm. Compute the SE and the MDE at 5% significance and 80% power.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Cluster-mean variance factor: $\\rho + (1 - \\rho)/m = 0.1 + 0.9/30 = 0.13$.\n", "\n", "**Step 2.** $\\operatorname{Var}(\\hat\\tau) = 0.13 \\times (1/30 + 1/30) = 0.13 \\times 0.0667 \\approx 0.00867$; SE $\\approx 0.093$.\n", "\n", "**Step 3.** MDE $= 2.80 \\times 0.093 \\approx 0.26$ sd.\n", "\n", "**Answer:** SE ≈ 0.093 sd; MDE ≈ 0.26 sd.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · More pupils or more schools?\n", "\n", "Starting from Exercise 3, compare (a) doubling pupils per school to 60 and (b) doubling schools to 60 per arm.\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a) Factor $0.1 + 0.9/60 = 0.115$; variance $0.115 \\times 0.0667 \\approx 0.00767$; SE $\\approx 0.088$; MDE $\\approx 0.245$.\n", "\n", "**Step 2.** (b) Factor 0.13; variance $0.13 \\times (2/60) \\approx 0.00433$; SE $\\approx 0.066$; MDE $\\approx 0.18$.\n", "\n", "**Answer:** both double the sample, but more schools cuts the MDE by 29% versus 6% for more pupils per school.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Estimating the ICC\n", "\n", "Between-clinic variance of patient satisfaction is 0.2 and within-clinic variance 0.8. Compute the ICC.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\rho = 0.2/(0.2 + 0.8) = 0.2$.\n", "\n", "**Answer:** 0.2: two patients of the same clinic have correlated satisfaction scores.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Variance of a cluster mean\n", "\n", "Under $Y_{ig} = \\mu + u_g + e_{ig}$ with independent components, prove $\\operatorname{Var}(\\bar Y_g) = \\sigma^2[\\rho + (1 - \\rho)/m]$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\bar Y_g = \\mu + u_g + \\bar e_g$, where $\\bar e_g$ averages $m$ independent errors.\n", "\n", "**Step 2.** $\\operatorname{Var}(\\bar Y_g) = \\sigma_u^2 + \\sigma_e^2/m$.\n", "\n", "**Step 3.** Substitute $\\sigma_u^2 = \\rho\\sigma^2$ and $\\sigma_e^2 = (1 - \\rho)\\sigma^2$: $\\sigma^2[\\rho + (1 - \\rho)/m]$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · The design effect\n", "\n", "Show that $\\sigma^2[\\rho + (1 - \\rho)/m] = [1 + (m - 1)\\rho]\\,\\sigma^2/m$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Factor out $\\sigma^2/m$: $\\sigma^2[\\rho + (1 - \\rho)/m] = \\frac{\\sigma^2}{m}[m\\rho + 1 - \\rho]$.\n", "\n", "**Step 2.** $m\\rho + 1 - \\rho = 1 + (m - 1)\\rho$. $\\blacksquare$\n", "\n", "The second form is \"individual-level variance $\\sigma^2/m$\" times the DEFF.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · The ceiling on cluster size\n", "\n", "Prove that as $m \\to \\infty$ with $J_1, J_0$ fixed, $\\operatorname{Var}(\\hat\\tau) \\to \\sigma^2\\rho(1/J_1 + 1/J_0)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $(1 - \\rho)/m \\to 0$, so the cluster-mean variance tends to $\\rho\\sigma^2$.\n", "\n", "**Step 2.** Multiply by $1/J_1 + 1/J_0$. $\\blacksquare$\n", "\n", "Even infinitely large clusters leave the between-cluster noise: only more clusters remove it.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Cluster means analysis\n", "\n", "With equal cluster sizes, show that the individual-level difference in means equals the difference in the averages of cluster means. What changes with unequal sizes?\n", "\n", "
Show solution\n", "\n", "**Step 1.** With $m$ pupils per cluster, the treated individual mean is $\\frac{1}{mJ_1}\\sum_g\\sum_iY_{ig} = \\frac{1}{J_1}\\sum_g\\bar Y_g$, the average of cluster means. The same holds for controls.\n", "\n", "**Step 2.** With unequal sizes, the individual mean weights clusters by size, while the average of cluster means weights them equally.\n", "\n", "**Answer:** they then estimate different estimands: the effect for the average *person* versus the average *cluster*. Pre-specify which one you want.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Ignoring clustering\n", "\n", "Show that an individual-level SE that ignores clustering is too small by the factor $\\sqrt{\\text{DEFF}}$ (equal cluster sizes, large samples).\n", "\n", "
Show solution\n", "\n", "**Step 1.** The naive formula treats all $mJ$ observations as independent: $\\sigma^2(1/(mJ_1) + 1/(mJ_0))$.\n", "\n", "**Step 2.** The true variance is DEFF times that (Exercise 7 and the theory block).\n", "\n", "**Step 3.** Ratio of SEs: $\\sqrt{\\text{DEFF}}$. $\\blacksquare$\n", "\n", "With DEFF = 3.9, naive SEs are about half the correct size and \"significant\" results can be spurious.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Planning a teacher-training trial\n", "\n", "Teacher training is randomised across schools. Pupils' test scores are standardised, $m = 25$ pupils per school, ICC = 0.15. How many schools are needed for an MDE of 0.20 sd with 5% significance, 80% power and half the schools treated?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Variance factor: $\\rho + (1 - \\rho)/m = 0.15 + 0.85/25 = 0.184$.\n", "\n", "**Step 2.** With $P = 0.5$, $P(1 - P)J = J/4$, so $\\text{MDE} = 2.80\\sqrt{0.184 \\times 4/J}$.\n", "\n", "**Step 3.** Set equal to 0.20: $(0.20/2.80)^2 = 0.736/J$, i.e. $0.0051 = 0.736/J$.\n", "\n", "**Step 4.** $J = 0.736/0.0051 \\approx 144.3$; round up to an even number.\n", "\n", "**Answer:** about 146 schools (73 per arm), 3,650 pupils.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · The value of a baseline test\n", "\n", "In Exercise 11, school-average baseline scores explain 50% of the between-school variance. If you adjust for them, roughly how many schools are needed?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Between-school component falls from 0.15 to 0.075; the within-school part stays at 0.85/25 = 0.034 (ignoring individual-level adjustment).\n", "\n", "**Step 2.** New factor: $0.075 + 0.034 = 0.109$.\n", "\n", "**Step 3.** $J = 0.109 \\times 4/0.0051 \\approx 85.5$, so about 86 schools.\n", "\n", "**Answer:** about 86 schools instead of 146: a cheap baseline test saves 60 schools.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Clinics of different sizes\n", "\n", "A trial randomises 40 clinics, from 20 to 500 patients each, to a new hypertension protocol. The protocol may work better in large clinics. Which estimand does a patient-level comparison target, and which does a comparison of clinic means target?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Patient-level: each patient counts equally, so large clinics dominate: the effect for the average patient.\n", "\n", "**Step 2.** Clinic-level: each clinic counts equally: the effect for the average clinic.\n", "\n", "**Step 3.** If effects differ by size, these differ. A health ministry rolling out across clinics may care about patients; a manager supporting clinics may care about clinics.\n", "\n", "**Answer:** state the estimand in the pre-analysis plan and weight accordingly.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Why not randomise individuals?\n", "\n", "A village-level information campaign on safe water could instead be randomised by household. Give two reasons to randomise villages.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Spillovers: households talk to neighbours, so control households in treated villages would be partly treated, biasing the comparison toward zero (Day 32).\n", "\n", "**Step 2.** Delivery: the campaign uses village meetings and posters, which cannot be withheld from some households.\n", "\n", "**Answer:** cluster randomisation trades statistical power for a clean, policy-relevant comparison.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Few clusters\n", "\n", "A district-level health reform is randomised across 8 districts (4 treated). What inference is appropriate?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Cluster-robust SEs need many clusters; with 8 they are unreliable (Day 29).\n", "\n", "**Step 2.** Randomisation inference at the district level: $\\binom84 = 70$ assignments, smallest two-sided p-value $2/70 \\approx 0.029$ (Day 37).\n", "\n", "**Step 3.** Alternatively, analyse district means with a t-test on 6 degrees of freedom, or the wild cluster bootstrap.\n", "\n", "**Answer:** use district-level randomisation inference, and recognise that only large effects can be detected with 8 districts.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from scipy import stats\n", "\n", "za, zb = stats.norm.ppf(0.975), stats.norm.ppf(0.80)\n", "def mde(rho, m, J, P=0.5, sigma=1.0):\n", " return (za + zb) * np.sqrt(sigma**2 * (rho + (1 - rho) / m) / (P * (1 - P) * J))\n", "\n", "print(\"Exercise 3 MDE:\", round(mde(0.1, 30, 60), 3))\n", "print(\"more pupils:\", round(mde(0.1, 60, 60), 3), \" more schools:\", round(mde(0.1, 30, 120), 3))\n", "J = 4 * 0.184 / (0.20 / (za + zb))**2\n", "print(\"schools needed (Ex 11):\", round(J, 1))\n", "\n", "# Simulate a cluster RCT: naive versus correct SE\n", "rng = np.random.default_rng(40)\n", "J, m, rho = 60, 30, 0.1\n", "est, naive_se = [], []\n", "for _ in range(3000):\n", " u = rng.normal(0, np.sqrt(rho), J); T = np.repeat(rng.permutation(np.repeat([0, 1], J // 2)), m)\n", " y = np.repeat(u, m) + rng.normal(0, np.sqrt(1 - rho), J * m) + 0.0 * T\n", " est.append(y[T == 1].mean() - y[T == 0].mean())\n", " naive_se.append(np.sqrt(y[T == 1].var(ddof=1) / (J * m / 2) + y[T == 0].var(ddof=1) / (J * m / 2)))\n", "print(f\"true SD {np.std(est):.4f}, mean naive SE {np.mean(naive_se):.4f}, ratio {np.std(est) / np.mean(naive_se):.2f}\"\n", " f\" (sqrt DEFF = {np.sqrt(1 + 29 * 0.1):.2f})\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Clustering inflates the variance by DEFF = 1 + (m − 1)ρ; ignoring it makes SEs too small by √DEFF.\n", "- Power comes mainly from the number of clusters; baseline covariates at the cluster level are very valuable.\n", "- With unequal cluster sizes, choose the estimand; with few clusters, use cluster-level randomisation inference." ] } ] }