{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 80: Phase 5 Capstone: Choosing a Design\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Instruments, differences-in-differences, discontinuities and bounds answer different questions under different assumptions. How to match a design to the variation available, what each one estimates, which diagnostics make it credible, and how designs can be combined.\n", "\n", "**Learning outcomes**\n", "\n", "- Match a policy question and its data to IV, DiD, RD, RKD or bounds, and name the estimand each delivers.\n", "- List the identifying assumption and the main diagnostics for each design.\n", "- Combine designs (Wald-DiD, difference in discontinuities) and prove when they work.\n", "- Explain why credible designs can give different numbers for the \"same\" effect." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Start from the variation\n", "\n", "The first question is not \"which estimator?\" but \"what moves treatment for reasons unrelated to outcomes?\"\n", "\n", "| Source of variation | Design | Estimand | Key assumption | Diagnostics |\n", "|---|---|---|---|---|\n", "| You randomise | RCT (Phase 3) | ATE (or ITT) | randomisation, no attrition bias | balance, attrition, compliance |\n", "| A random or as-good-as-random nudge to take-up | IV (61–66) | LATE for compliers | independence, exclusion, monotonicity | first stage F, balance on Z, AR intervals, complier profile |\n", "| A policy changes for some groups at some time | DiD / event study (67–72) | ATT for treated groups | parallel trends, no anticipation | pre-trends, placebo outcomes, sensitivity bounds, robust staggered estimators |\n", "| A rule assigns treatment at a threshold | RD (73–77) | effect at the cutoff (compliers there if fuzzy) | continuity at the cutoff | density test, covariate balance, placebo cutoffs, bandwidth sensitivity |\n", "| A rule changes the slope of a schedule | RKD (78) | marginal effect at the kink | smoothness | slope placebo, density derivative |\n", "| None of the above is credible | bounds (79), sensitivity (95) | identified set | only what you state | transparency about assumptions |\n", "\n", "### Estimands differ\n", "\n", "Each design answers a question about a particular group: compliers with a lottery, treated states, students at a test-score cutoff. With heterogeneous effects, credible designs applied to the same policy can legitimately disagree. Before comparing numbers, compare estimands.\n", "\n", "### Combining designs\n", "\n", "- **Wald-DiD (fuzzy DiD):** if a policy raises take-up in treated groups, divide the DiD in outcomes by the DiD in take-up.\n", "- **Difference in discontinuities:** if a jump already existed at a cutoff before a policy started applying there, subtract the old jump from the new one.\n", "- **RD + bounds, IV + DiD, event studies with instruments**: assumptions combine, and so do diagnostics.\n", "\n", "### A workflow\n", "\n", "1. State the causal question and the estimand (whose effect, of what, on what, when).\n", "2. Describe the assignment mechanism with institutional detail; draw the DAG.\n", "3. Choose the design that matches the variation; state its assumptions plainly.\n", "4. Run the diagnostics; report the reduced form or raw comparisons graphically.\n", "5. Report estimates with robust inference and sensitivity analyses.\n", "6. Discuss external validity: for whom does the estimate speak?\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Match the design\n", "\n", "Name the design: (a) A city lets residents enter a lottery for housing vouchers. (b) A province raises its minimum wage; neighbouring provinces do not. (c) Children born after 1 September start school a year later. (d) A disability benefit replaces 60% of earnings up to a cap. (e) An NGO enrols villages based on undocumented criteria, with no comparison group.\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a) IV (lottery as instrument; ITT plus LATE).\n", "\n", "**Step 2.** (b) DiD / event study.\n", "\n", "**Step 3.** (c) RD on birth date (fuzzy, as some parents defer).\n", "\n", "**Step 4.** (d) RKD at the cap.\n", "\n", "**Step 5.** (e) Bounds or selection on observables with sensitivity analysis; no clean design.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · What can be checked?\n", "\n", "For IV, DiD and RD, name one implication of the key assumption that can be checked in data and one part that cannot.\n", "\n", "
Show solution\n", "\n", "**Step 1.** IV: balance of pre-determined covariates across instrument values (checkable); exclusion (not checkable).\n", "\n", "**Step 2.** DiD: parallel pre-trends (checkable); parallel post-treatment trends (not).\n", "\n", "**Step 3.** RD: density and covariate continuity (checkable); continuity of potential outcomes themselves (not directly).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Three numbers for one programme\n", "\n", "A school-meals programme is evaluated three ways: an RD at the eligibility income cutoff (+0.12 sd in test scores), a DiD across districts that adopted early (+0.08), and a lottery among oversubscribed schools (+0.15 per pupil fed). Should they agree?\n", "\n", "
Show solution\n", "\n", "**Step 1.** RD: effect for pupils at the income cutoff.\n", "\n", "**Step 2.** DiD: ATT for pupils in early-adopting districts (a mix of incomes).\n", "\n", "**Step 3.** Lottery IV: LATE for pupils in oversubscribed schools who ate because their school won.\n", "\n", "**Answer:** not necessarily; with heterogeneous effects they estimate different averages. Disagreement is informative about heterogeneity, provided each design passes its diagnostics.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Name the estimand\n", "\n", "Give the estimand of each: a sharp RD; a fuzzy RD; a Wald estimator with one-sided non-compliance; a 2×2 DiD; Lee bounds.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Sharp RD: ATE at the cutoff.\n", "\n", "**Step 2.** Fuzzy RD: LATE for compliers at the cutoff.\n", "\n", "**Step 3.** One-sided Wald: ATT (effect on the treated).\n", "\n", "**Step 4.** 2×2 DiD: ATT for the treated group in the post period.\n", "\n", "**Step 5.** Lee bounds: an interval for the effect on always-observed units.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · A diagnostics list\n", "\n", "For a staggered DiD of a state paid-leave law, list four diagnostics.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Event-study pre-trends using a heterogeneity-robust estimator.\n", "\n", "**Step 2.** Bacon decomposition of TWFE to see forbidden comparisons.\n", "\n", "**Step 3.** Placebo outcomes (groups not covered by the law).\n", "\n", "**Step 4.** Sensitivity to bounded violations of parallel trends (breakdown value).\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Wald-DiD\n", "\n", "Suppose a reform raises take-up $D$ more in group 1 than in group 0, and $Y = \\alpha_g + \\lambda_t + \\tau D + \\varepsilon$ with a constant effect $\\tau$ and parallel trends in $\\varepsilon$. Prove that the DiD in $Y$ divided by the DiD in $D$ equals $\\tau$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Group-period means: $\\bar Y_{gt} = \\alpha_g + \\lambda_t + \\tau\\bar D_{gt} + \\bar\\varepsilon_{gt}$.\n", "\n", "**Step 2.** DiD in $Y$: the $\\alpha_g$ and $\\lambda_t$ cancel; parallel trends cancels the $\\varepsilon$ terms; left with $\\tau\\times$ DiD in $D$.\n", "\n", "**Step 3.** Divide (DiD in $D$ non-zero). $\\blacksquare$ With heterogeneous effects, extra assumptions are needed for a LATE interpretation (de Chaisemartin and D'Haultfœuille).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Difference in discontinuities\n", "\n", "A cutoff at age 65 had a jump $J_0$ in employment before a pension reform (from other age-65 rules) and $J_1$ after the reform also started at 65. Show that $J_1 - J_0$ identifies the reform's effect at 65 if the other rules' jump is unchanged.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $J_0 = \\delta$ (other rules); $J_1 = \\delta + \\tau$ (other rules plus reform).\n", "\n", "**Step 2.** $J_1 - J_0 = \\tau$ when $\\delta$ is the same in both periods. $\\blacksquare$ This combines RD's local comparison with DiD's removal of a stable confounder.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Placebo outcomes\n", "\n", "Show that under parallel trends, a DiD on an outcome that treatment cannot affect has expectation zero, and explain what a non-zero placebo implies.\n", "\n", "
Show solution\n", "\n", "**Step 1.** For an unaffected outcome, $Y(1) = Y(0)$, so the ATT is zero.\n", "\n", "**Step 2.** By Day 67, Exercise 9, the DiD equals ATT + trend bias = trend bias.\n", "\n", "**Answer:** if parallel trends holds for the placebo, its DiD is zero in expectation; a non-zero placebo reveals differential trends, which may also affect the real outcome (if the outcomes share trend drivers). $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · OLS or IV by mean squared error\n", "\n", "OLS has bias $b$ and variance $v_O$; IV is unbiased with variance $v_I > v_O$. Show IV has lower MSE iff $b^2 > v_I - v_O$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\text{MSE}_{OLS} = b^2 + v_O$; $\\text{MSE}_{IV} = v_I$.\n", "\n", "**Step 2.** $v_I < b^2 + v_O \\iff b^2 > v_I - v_O$. $\\blacksquare$ With a weak instrument, $v_I$ can be so large that a biased OLS is closer to the truth, but its bias is unknown, so IV's confidence interval is the honest summary.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Reweighting for external validity\n", "\n", "Suppose effects vary only with a covariate $X$: $\\tau(x)$, and a study identifies $\\tau(x)$ for each $x$ in its sample. Show the ATE for a target population with covariate distribution $Q$ is $\\int\\tau(x)\\,dQ(x)$, and that the study's own average uses its own distribution $P$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Target ATE: $E_Q[Y(1) - Y(0)] = E_Q[E[Y(1) - Y(0) \\mid X]] = E_Q[\\tau(X)]$.\n", "\n", "**Step 2.** The study's average is $E_P[\\tau(X)]$; they differ when $P \\ne Q$ and $\\tau$ varies.\n", "\n", "**Answer:** reweighting the study's conditional effects to the target's covariate mix transports the estimate, if effects depend only on observed $X$ (and $Q$'s support is covered). $\\blacksquare$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Minimum wages: two designs\n", "\n", "One study compares contiguous border counties across state lines (DiD with county pairs); another uses the bunching of wages at the new minimum to count jobs lost below it. What does each assume, and why might they disagree?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Border-pair DiD: parallel trends between neighbouring counties; estimates employment effects for those counties.\n", "\n", "**Step 2.** Bunching: jobs that disappear below the new minimum versus excess jobs at and above it, relative to a counterfactual wage distribution; assumes the counterfactual distribution can be predicted (e.g., from other states).\n", "\n", "**Answer:** they use different variation and estimands; disagreement may come from spillovers up the wage distribution or from local labour-market differences. Each paper's diagnostics, not its design label, decide credibility.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Class size: RD versus an experiment\n", "\n", "The Tennessee STAR experiment found that classes about 7 pupils smaller raised scores by about 0.2 sd; Maimonides' rule RD finds about 0.25 points per pupil on a different test in Israel. How should these be compared?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Put both in comparable units (sd per pupil): STAR ≈ 0.03 sd per pupil; convert the RD estimate using the test's sd.\n", "\n", "**Step 2.** Estimands differ: STAR's ATE for kindergarten to grade 3 in Tennessee; the RD's effect at the 40-pupil threshold in Israeli schools.\n", "\n", "**Answer:** agreement in standardised units across designs increases confidence; differences may reflect age, baseline class size and context rather than bias.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Compulsory schooling: IV or RD?\n", "\n", "Evaluate two designs for the effect of an extra year of schooling: (a) IV using changes in compulsory-schooling laws across states and cohorts; (b) RD on school-entry birth-date cutoffs.\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a) LATE for students who stay in school only because of the law (likely low-attainment); threats: other policies changing with the laws, weak first stages.\n", "\n", "**Step 2.** (b) Compares children born just before and after the cutoff: they differ in school-starting age and in age at test, not only in years of schooling; exclusion is problematic for \"years of schooling\".\n", "\n", "**Answer:** (a) speaks to compulsory schooling policy for dropouts; (b) is better suited to the effect of school-starting age.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · A staggered hospital payment reform\n", "\n", "A country moves hospitals from fee-for-service to bundled payments, region by region over four years, plus one region that never switches. Design the evaluation.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Estimand: ATT on cost per episode and readmissions, by years since adoption.\n", "\n", "**Step 2.** Design: group-time ATTs (Callaway–Sant'Anna) with never-treated and not-yet-treated regions; event-study aggregation over a balanced window.\n", "\n", "**Step 3.** Diagnostics: pre-trends, placebo outcomes (conditions not covered by bundles), Bacon decomposition of any TWFE estimate, sensitivity to bounded violations.\n", "\n", "**Step 4.** Inference clustered by region (few clusters: wild bootstrap or randomisation inference over adoption dates).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · One world, three estimands\n", "\n", "In the simulation below, a programme's effect is larger for poorer households: $\\tau(x) = 2 - 1.5x$ for a poverty score $x \\in [-1, 1]$ (lower is poorer). Households with $x < 0$ are eligible; a lottery among eligible households offers places, and poorer households comply more. Compute the population ATE, the ATE among eligible households, the RD estimate at the eligibility cutoff (using a hypothetical universal take-up rule), and the lottery's LATE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Population ATE: $2 - 1.5E[x] = 2$.\n", "\n", "**Step 2.** Eligible ATE: $2 - 1.5(-0.5) = 2.75$.\n", "\n", "**Step 3.** RD at the cutoff: $\\tau(0) = 2$.\n", "\n", "**Step 4.** Lottery LATE: compliers are poorer than the average eligible household, so the LATE exceeds 2.75 (the code finds about 2.9).\n", "\n", "**Answer:** four correct numbers for four questions. Policy makers need the one matching the decision: expanding eligibility (RD-type), scaling up within the eligible (eligible ATE), or improving outreach (LATE-type).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(80); n = 2_000_000\n", "x = rng.uniform(-1, 1, n)\n", "tau = 2 - 1.5 * x\n", "y0 = 5 + x + rng.normal(0, 1, n)\n", "eligible = x < 0\n", "print(f\"population ATE {tau.mean():.3f} eligible ATE {tau[eligible].mean():.3f}\")\n", "\n", "# Sharp RD at the cutoff with universal take-up for the eligible\n", "d_rd = eligible.astype(float); y = y0 + d_rd * tau\n", "def side(xx, yy, h=.2):\n", " w = np.clip(1 - np.abs(xx) / h, 0, None); m = w > 0\n", " X = np.column_stack([np.ones(m.sum()), xx[m]]); W = w[m]\n", " return np.linalg.solve(X.T @ (W[:, None] * X), X.T @ (W * yy[m]))[0]\n", "rd = side(-x[eligible], y[eligible]) - side(x[~eligible], y[~eligible])\n", "print(f\"RD at cutoff {rd:.3f} (tau(0) = 2)\")\n", "\n", "# Lottery among the eligible; poorer households comply more\n", "e = eligible\n", "Z = rng.random(n) < .5\n", "comply = rng.random(n) < (0.3 - 0.6 * x) # poorer households (x near -1) comply with prob 0.9, those near 0 with 0.3\n", "D = e & Z & comply\n", "yL = y0 + D * tau\n", "wald = (yL[e & Z].mean() - yL[e & ~Z].mean()) / (D[e & Z].mean() - D[e & ~Z].mean())\n", "print(f\"lottery LATE {wald:.3f} (true complier average {tau[e & comply].mean():.3f})\")\n", "\n", "# Exercise 9: OLS vs IV by MSE\n", "b, vO, vI = .05, .0001, .0016\n", "print(\"IV preferred by MSE:\", b ** 2 > vI - vO)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Start from the source of exogenous variation; it determines the design, its assumptions and its estimand.\n", "- Every design has testable implications; run them, plot the raw comparisons and report robust inference.\n", "- Credible designs can disagree because they target different people; designs can also be combined (Wald-DiD, difference in discontinuities) when one alone is not enough." ] } ] }