{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 39: Regression Adjustment in Experiments (Lin's Estimator)\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Using baseline covariates to sharpen experimental estimates, Freedman's critique of ANCOVA, and Lin's fully interacted fix.\n", "\n", "**Learning outcomes**\n", "\n", "- Write the ANCOVA and Lin estimators in terms of group means and slopes.\n", "- Prove the adjustment formulas and the link between Lin's estimator and post-stratification.\n", "- Explain Freedman's critique and why the interacted regression is asymptotically never worse than the difference in means.\n", "- Choose covariates for adjustment and avoid specification searching." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Why adjust?\n", "\n", "Randomisation already makes the difference in means unbiased. Adjusting for a baseline covariate $X$ that predicts the outcome removes outcome noise (Day 27, Exercise 14) and corrects chance imbalance (Day 35, Exercise 9).\n", "\n", "### ANCOVA\n", "\n", "Regress $Y$ on an intercept, $T$ and $X$:\n", "\n", "$$\n", "\\hat\\tau_{\\text{ANCOVA}} = (\\bar Y_1 - \\bar Y_0) - \\hat\\beta(\\bar X_1 - \\bar X_0),\n", "$$\n", "\n", "where $\\hat\\beta$ is the pooled within-arm slope. The raw difference is corrected for the covariate imbalance.\n", "\n", "### Freedman's critique\n", "\n", "Freedman (2008) showed that, under randomisation alone, ANCOVA is biased in small samples (bias of order $1/n$) and can be *less* precise than the raw difference when treatment effects vary with $X$ and the arms have unequal sizes.\n", "\n", "### Lin's estimator\n", "\n", "Lin (2013) answered: interact treatment with the **centred** covariate,\n", "\n", "$$\n", "Y = \\alpha + \\tau T + \\beta(X - \\bar X) + \\gamma\\,T(X - \\bar X) + e,\n", "$$\n", "\n", "and use heteroskedasticity-robust SEs. Equivalently, fit separate regressions in each arm (slopes $\\hat\\beta_1, \\hat\\beta_0$) and compare predictions at the overall mean $\\bar X$:\n", "\n", "$$\n", "\\hat\\tau_{\\text{Lin}} = \\big[\\bar Y_1 - \\hat\\beta_1(\\bar X_1 - \\bar X)\\big] - \\big[\\bar Y_0 - \\hat\\beta_0(\\bar X_0 - \\bar X)\\big].\n", "$$\n", "\n", "Properties: consistent; asymptotically at least as precise as the difference in means *and* as ANCOVA; robust SEs are (asymptotically) conservative. With a categorical $X$, it is exactly the post-stratified estimator (Day 38).\n", "\n", "### Good practice\n", "\n", "Pre-specify a small number of strong baseline predictors (above all, the baseline value of the outcome). Never choose covariates after looking at the estimated effects.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · ANCOVA by hand\n", "\n", "In a job-training RCT, treated workers earn \\$1,200 more at follow-up, but they also earned \\$500 more at baseline. The pooled within-arm slope of follow-up on baseline earnings is 0.8. Compute the ANCOVA estimate.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Correction: $\\hat\\beta(\\bar X_1 - \\bar X_0) = 0.8 \\times 500 = 400$.\n", "\n", "**Step 2.** $\\hat\\tau_{\\text{ANCOVA}} = 1{,}200 - 400 = 800$.\n", "\n", "**Answer:** \\$800: one-third of the raw gap reflected the baseline head start.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Lin by hand\n", "\n", "Test scores: $\\bar Y_1 = 20$, $\\bar Y_0 = 17$; baseline $\\bar X_1 = 10.5$, $\\bar X_0 = 9.5$, $\\bar X = 10$; within-arm slopes $\\hat\\beta_1 = 1.2$, $\\hat\\beta_0 = 0.8$. Compute $\\hat\\tau_{\\text{Lin}}$ and compare with the raw difference.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Treated prediction at $\\bar X$: $20 - 1.2(10.5 - 10) = 20 - 0.6 = 19.4$.\n", "\n", "**Step 2.** Control prediction at $\\bar X$: $17 - 0.8(9.5 - 10) = 17 + 0.4 = 17.4$.\n", "\n", "**Step 3.** $\\hat\\tau_{\\text{Lin}} = 19.4 - 17.4 = 2.0$, versus the raw difference 3.\n", "\n", "**Answer:** 2.0. The treated group started ahead, and part of the raw gap was that head start.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Precision gain\n", "\n", "The baseline score explains 50% of the variance of end-line scores within each arm. Approximately how much does adjustment shrink the SE?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Residual variance falls to $(1 - 0.5)\\sigma^2$.\n", "\n", "**Step 2.** SE falls by $\\sqrt{0.5} \\approx 0.71$.\n", "\n", "**Answer:** about 29% smaller, worth doubling the sample (Day 27).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Lin with a binary covariate\n", "\n", "Urban (60% of households) and rural (40%). Effects of a cash transfer estimated within each: urban 10, rural 25. What does Lin's estimator with an urban dummy give?\n", "\n", "
Show solution\n", "\n", "**Step 1.** With a binary $X$, Lin's estimator is the post-stratified estimator (Exercise 8).\n", "\n", "**Step 2.** $0.6(10) + 0.4(25) = 6 + 10 = 16$.\n", "\n", "**Answer:** 16.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Which covariates?\n", "\n", "An RCT of a reading programme measures: baseline reading score, age, gender, classroom noise during the programme, and end-line attendance. Which should go in the adjustment?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Baseline reading score: strongest predictor, measured before randomisation: yes.\n", "\n", "**Step 2.** Age and gender: pre-treatment, modestly predictive: acceptable if pre-specified.\n", "\n", "**Step 3.** Noise during the programme and end-line attendance: measured after assignment, possibly affected by treatment: no.\n", "\n", "**Answer:** baseline score (plus age and gender if pre-specified).\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · The ANCOVA formula\n", "\n", "Prove $\\hat\\tau_{\\text{ANCOVA}} = (\\bar Y_1 - \\bar Y_0) - \\hat\\beta(\\bar X_1 - \\bar X_0)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The regressors include an intercept and $T$, whose span contains both arm indicators. Residuals are orthogonal to them, so they average to zero within each arm.\n", "\n", "**Step 2.** Averaging the fitted equation within the treated arm: $\\bar Y_1 = \\hat\\alpha + \\hat\\tau + \\hat\\beta\\bar X_1$.\n", "\n", "**Step 3.** Within the control arm: $\\bar Y_0 = \\hat\\alpha + \\hat\\beta\\bar X_0$.\n", "\n", "**Step 4.** Subtract: $\\bar Y_1 - \\bar Y_0 = \\hat\\tau + \\hat\\beta(\\bar X_1 - \\bar X_0)$. Rearrange. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Lin = separate regressions\n", "\n", "Show that the interacted regression gives the same fitted values as separate regressions of $Y$ on $X$ in each arm, and that $\\hat\\tau$ is the difference of the two fitted lines at $\\bar X$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The four regressors $1, T, (X - \\bar X), T(X - \\bar X)$ span the same space as $(1 - T), T, (1 - T)(X - \\bar X), T(X - \\bar X)$: an intercept and slope for each arm.\n", "\n", "**Step 2.** So OLS fits each arm separately: control line $\\hat\\alpha + \\hat\\beta(X - \\bar X)$, treated line $\\hat\\alpha + \\hat\\tau + (\\hat\\beta + \\hat\\gamma)(X - \\bar X)$.\n", "\n", "**Step 3.** At $X = \\bar X$, the lines are $\\hat\\alpha$ and $\\hat\\alpha + \\hat\\tau$, so $\\hat\\tau$ is their difference.\n", "\n", "**Step 4.** Each arm's line passes through its means, so its value at $\\bar X$ is $\\bar Y_t - \\hat\\beta_t(\\bar X_t - \\bar X)$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Lin = post-stratification for a categorical covariate\n", "\n", "With $X$ a set of category dummies (centred), show $\\hat\\tau_{\\text{Lin}} = \\sum_g(N_g/N)\\hat\\tau_g$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Separate regressions on category dummies in each arm fit the category-by-arm means $\\bar Y_{tg}$ (Day 20, Exercise 14).\n", "\n", "**Step 2.** \"Evaluating at the overall mean of the centred dummies\" means averaging category predictions with weights $N_g/N$.\n", "\n", "**Step 3.** So $\\hat\\tau_{\\text{Lin}} = \\sum_g\\frac{N_g}{N}\\bar Y_{1g} - \\sum_g\\frac{N_g}{N}\\bar Y_{0g} = \\sum_g\\frac{N_g}{N}\\hat\\tau_g$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Why ANCOVA can hurt\n", "\n", "Explain, using Exercise 6, why ANCOVA's pooled slope can make it less precise than the raw difference when the arms are unbalanced (e.g. 80/20) and slopes differ between arms.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The ideal correction for the treated mean uses $\\beta_1$, and for the control mean uses $\\beta_0$.\n", "\n", "**Step 2.** ANCOVA uses one pooled slope, which with 80% treated is close to $\\beta_1$, but the variance of $\\hat\\tau$ is dominated by the *smaller* arm (20% controls), whose correction then uses the wrong slope.\n", "\n", "**Step 3.** With $\\beta_1$ and $\\beta_0$ far apart, the mis-corrected control mean can be noisier than the unadjusted one.\n", "\n", "**Answer:** separate slopes (Lin) fix this; asymptotically Lin's variance is never larger than the difference in means' (Lin, 2013).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Small-sample bias\n", "\n", "Lin's estimator is consistent but not exactly unbiased. Why does estimating slopes introduce bias of order $1/n$, and why does this rarely matter?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\hat\\beta_t$ is estimated from the same data as $\\bar X_t$ and $\\bar Y_t$; the product $\\hat\\beta_t(\\bar X_t - \\bar X)$ has a small correlation-induced bias.\n", "\n", "**Step 2.** Each factor has error of order $1/\\sqrt n$, so the product's bias is of order $1/n$.\n", "\n", "**Step 3.** The SE is of order $1/\\sqrt n$, so the bias is negligible relative to the SE except in very small experiments.\n", "\n", "**Answer:** in very small trials, prefer the difference in means or randomisation inference with the adjusted statistic (Day 37).\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Training and baseline earnings\n", "\n", "A training RCT (500 per arm) reports: raw difference \\$1,000 (SE 600); ANCOVA \\$850 (SE 420); Lin \\$840 (SE 410). Which should be the headline, and why are the estimates similar?\n", "\n", "
Show solution\n", "\n", "**Step 1.** With equal arms and a strong pre-treatment predictor, all three are consistent for the ATE; the point estimates differ only through chance imbalance.\n", "\n", "**Step 2.** Adjustment cuts the SE by about 30%: the raw estimate is not significant ($z = 1.67$), Lin's is ($z \\approx 2.05$).\n", "\n", "**Answer:** if pre-specified, report Lin's estimate (\\$840, SE 410) as the headline and show the raw difference too.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Specification searching\n", "\n", "A team tries 20 combinations of covariates and reports the one with the smallest p-value. Why is this a problem even though each regression is valid?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Each specification gives a slightly different estimate; choosing the most significant is a multiple-testing problem (Day 10).\n", "\n", "**Step 2.** The reported p-value ignores the search, so the false-positive rate exceeds 5%.\n", "\n", "**Answer:** pre-register the covariates (or use a pre-specified rule such as the baseline outcome only), and report the raw difference as a robustness check.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Too many covariates\n", "\n", "A trial with 60 participants adjusts for 25 baseline variables. What can go wrong, and what is a better plan?\n", "\n", "
Show solution\n", "\n", "**Step 1.** With $k$ close to $n$, the slopes are noisy, the $1/n$ bias grows, and robust SEs become unreliable.\n", "\n", "**Step 2.** Lin's estimator with interactions doubles the number of parameters, making it worse.\n", "\n", "**Answer:** adjust for 1–3 strong pre-specified predictors (the baseline outcome first), or a pre-registered summary index; or block on them at the design stage (Day 38).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Adjustment with a binary outcome\n", "\n", "A vaccination-reminder RCT measures uptake (0/1) and adjusts for whether the person was vaccinated last season using a linear probability model. Is the adjusted coefficient still the ATE on uptake?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Lin's results need no outcome model: the regression is just a device for combining arm means and slopes.\n", "\n", "**Step 2.** With a binary covariate, Lin's estimator equals the post-stratified difference in uptake (Exercise 8), an average of risk differences.\n", "\n", "**Answer:** yes; use robust SEs because a linear probability model is heteroskedastic (Day 29).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · An unbalanced design\n", "\n", "A costly treatment is given to 20% of 1,000 firms; effects on sales grow with firm size (different slopes by arm). In simulation (code below), compare the SDs of the raw difference, ANCOVA and Lin.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Run many randomisations of the same firms and compute the three estimators each time.\n", "\n", "**Step 2.** The simulation shows Lin's estimator has the smallest SD, the raw difference the largest; ANCOVA sits in between and is clearly worse than Lin.\n", "\n", "**Answer:** with unbalanced arms and heterogeneous slopes, use the interacted (Lin) specification.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(39)\n", "N, N1 = 1000, 200\n", "x = rng.lognormal(0, 0.6, N); xc = x - x.mean()\n", "y0 = 10 + 2 * xc + rng.normal(0, 2, N)\n", "y1 = y0 + 3 + 6 * xc # effect grows with firm size\n", "ate = (y1 - y0).mean()\n", "\n", "def ols(X, y):\n", " return np.linalg.lstsq(X, y, rcond=None)[0]\n", "\n", "raw, anc, lin = [], [], []\n", "for _ in range(4000):\n", " T = np.zeros(N, bool); T[rng.choice(N, N1, replace=False)] = True\n", " y = np.where(T, y1, y0)\n", " raw.append(y[T].mean() - y[~T].mean())\n", " anc.append(ols(np.column_stack([np.ones(N), T, xc]), y)[1])\n", " lin.append(ols(np.column_stack([np.ones(N), T, xc, T * xc]), y)[1])\n", "for name, e in ((\"raw difference\", raw), (\"ANCOVA\", anc), (\"Lin\", lin)):\n", " print(f\"{name:15s} mean {np.mean(e):.3f} (ATE {ate:.3f}) SD {np.std(e):.3f}\")\n", "\n", "print(\"Exercise 2 (Lin by hand):\", (20 - 1.2 * 0.5) - (17 - 0.8 * (-0.5)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Adjusting for strong baseline predictors corrects chance imbalance and cuts the SE.\n", "- ANCOVA can misbehave with unbalanced arms and heterogeneous effects; Lin's interacted regression with robust SEs is safe.\n", "- Pre-specify a few covariates; never search for the most significant specification." ] } ] }