{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 77: RD Validity: Density Tests & Covariate Balance\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "RD fails when units can sort precisely around the cutoff or when something else changes there. The checks that make an RD credible are a smooth density of the running variable, no jumps in pre-determined covariates, no effects at placebo cutoffs, and robustness to dropping observations right at the threshold.\n", "\n", "**Learning outcomes**\n", "\n", "- Explain why imprecise control over the running variable makes RD as good as a local experiment, and precise control breaks it.\n", "- Carry out and interpret a density (manipulation) test.\n", "- Use covariate balance, placebo cutoffs and donut RD as falsification checks.\n", "- Quantify the bias from sorting and diagnose it in health, education and political applications." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Why RD can work: imprecise control\n", "\n", "Lee (2008): if units influence their running variable but **cannot control it precisely** (there is a random component with a continuous density), then the distribution of their characteristics, and hence of potential outcomes, is continuous at the cutoff. Units just above and just below are then as good as randomly assigned. If units **can** place themselves precisely on the preferred side (teachers rounding scores up to a pass mark, firms adjusting reported size to stay under a regulation threshold), those just above differ systematically from those just below.\n", "\n", "### Test 1: the density of the running variable\n", "\n", "Under no manipulation, the density of $X$ is continuous at $c$. **McCrary (2008)** and **Cattaneo, Jansson and Ma** estimate the density on each side by local polynomial methods and test\n", "\n", "$$\n", "\\theta = \\ln f(c^+) - \\ln f(c^-) = 0.\n", "$$\n", "\n", "A spike just above $c$ and a dip just below indicate sorting. The test has no power against manipulation that moves units in both directions equally.\n", "\n", "### Test 2: covariate balance\n", "\n", "Run the RD with **pre-determined covariates** (measured before treatment, or fixed: sex, prior scores, parental income) as outcomes. They should not jump. A jump means the units on either side differ in more than treatment.\n", "\n", "### Test 3: placebo cutoffs and outcomes\n", "\n", "Estimate \"effects\" at fake cutoffs where nothing happens (for instance, the median of $X$ on each side), and effects on outcomes the treatment cannot affect. Both should be zero.\n", "\n", "### Robustness: donut RD\n", "\n", "If manipulation or **heaping** (values piling up at round numbers, such as birth weights recorded at 1,500 g) is concentrated very close to $c$, drop observations within a small \"donut\" around $c$ and re-estimate. Stability reassures; sensitivity reveals problems (at the cost of more extrapolation).\n", "\n", "### Other threats\n", "\n", "Compound treatments (several policies switch at the same age), and sorting on the outcome side (selection into the sample that depends on crossing the cutoff).\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · A crude density check\n", "\n", "Bins of equal width just below and just above a cutoff contain 480 and 620 observations. Compute the log density ratio and a rough z-statistic using $\\operatorname{SE} \\approx \\sqrt{1/n_1 + 1/n_2}$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\ln(620/480) = \\ln 1.292 \\approx 0.256$.\n", "\n", "**Step 2.** $\\operatorname{SE} \\approx \\sqrt{1/480 + 1/620} = \\sqrt{0.00208 + 0.00161} \\approx 0.061$.\n", "\n", "**Step 3.** $z \\approx 0.256/0.061 \\approx 4.2$.\n", "\n", "**Answer:** strong evidence of excess mass just above the cutoff. (A proper test fits the density's slope on each side; this bin comparison ignores it.)\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · A covariate jumps\n", "\n", "In an RD at a scholarship cutoff, the share of students with college-educated parents jumps by 6 points. What does this suggest?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Parental education is determined before the exam and cannot be caused by the scholarship.\n", "\n", "**Answer:** students on either side differ; likely sorting (re-takes, appeals) by better-resourced families. The outcome jump may partly reflect background, not the scholarship.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Placebo cutoffs\n", "\n", "At four placebo cutoffs, estimated jumps are 0.3, −0.5, 0.1 and 0.2 (each SE 0.4); at the true cutoff, 2.5 (SE 0.4). Interpret.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Placebo jumps are all within about 1.25 SE of zero.\n", "\n", "**Step 2.** The real jump is more than 6 SE.\n", "\n", "**Answer:** consistent with an effect specific to the cutoff, not a general wiggle in $E[Y \\mid X]$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Why a donut?\n", "\n", "Birth weights are often recorded at round numbers (1,500 g) and babies at those values have higher mortality for reasons unrelated to treatment. Why drop them?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Heaped observations are a different population (for example, poorer hospitals record rounded weights) concentrated exactly at the cutoff.\n", "\n", "**Step 2.** They distort the estimates of both limits at $c$.\n", "\n", "**Answer:** excluding a small window around $c$ checks whether the estimate is driven by them.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Limits of the density test\n", "\n", "Give an example where manipulation exists but the density test cannot detect it.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Some units push their score up across the cutoff while an equal number with different characteristics push theirs down (or are pushed down by an examiner).\n", "\n", "**Answer:** the density stays smooth but the composition changes; covariate balance tests can still detect it.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Imprecise control gives continuity\n", "\n", "Let $X = X^* + \\eta$, where units choose $X^*$ (possibly based on their potential outcomes) and $\\eta$ is independent noise with a continuous density $g$. Show that the density of $X$ given any type of unit is continuous, so the share of each type is continuous at $c$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** For a unit type $u$ with chosen $X^* = x^*_u$, the density of $X$ is $g(x - x^*_u)$, continuous in $x$.\n", "\n", "**Step 2.** Mixing over types: $f(x \\mid u)$ continuous, and $P(u \\mid X = x) = f(x \\mid u)P(u)/f(x)$ is continuous wherever $f(x) > 0$.\n", "\n", "**Step 3.** So the composition of units, and $E[Y(d) \\mid X = x]$, is continuous at $c$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · SE of a log count ratio\n", "\n", "With independent Poisson counts $n_1, n_2$, show $\\operatorname{Var}(\\ln n_2 - \\ln n_1) \\approx 1/n_1 + 1/n_2$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Delta method: $\\operatorname{Var}(\\ln n) \\approx \\operatorname{Var}(n)/E[n]^2 = E[n]/E[n]^2 = 1/E[n]$ for a Poisson count.\n", "\n", "**Step 2.** Independence: add the two variances. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · The bias from sorting\n", "\n", "Suppose a share $s$ of units just above the cutoff are \"movers\" who would naturally have been just below, with mean untreated outcome $\\mu_0 + \\Delta$ instead of $\\mu_0$, and the effect is $\\tau$ for everyone. Show the RD estimate converges to $\\tau + s\\Delta$ if movers come from a thin band just below (whose loss does not change the left limit).\n", "\n", "
Show solution\n", "\n", "**Step 1.** Right limit: $(1 - s)(\\mu_0 + \\tau) + s(\\mu_0 + \\Delta + \\tau) = \\mu_0 + \\tau + s\\Delta$.\n", "\n", "**Step 2.** Left limit: $\\mu_0$ (by assumption).\n", "\n", "**Step 3.** Difference: $\\tau + s\\Delta$. $\\blacksquare$ Positive selection of movers ($\\Delta > 0$) overstates the effect.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Covariate balance is implied by continuity\n", "\n", "Show that if the composition of units is continuous at $c$ (Exercise 6), then $E[W \\mid X = x]$ is continuous at $c$ for every pre-determined $W$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $E[W \\mid X = x] = \\sum_u E[W \\mid u]P(u \\mid X = x)$, with $W$ fixed for each unit type.\n", "\n", "**Step 2.** Each $P(u \\mid X = x)$ is continuous, so the sum is. $\\blacksquare$ A jump in $E[W \\mid X]$ therefore contradicts the continuity assumption that justifies the RD.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Two-sided manipulation with a smooth density\n", "\n", "Construct an example in which the density of $X$ is continuous at $c$ but the composition jumps.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Start with a uniform $X$ for two equal-sized groups, A and B.\n", "\n", "**Step 2.** A teacher moves A students from $[c - \\epsilon, c)$ to $[c, c + \\epsilon)$, and an equal number of B students from $[c, c + \\epsilon)$ to $[c - \\epsilon, c)$.\n", "\n", "**Step 3.** Total counts on each side are unchanged, but just above $c$ there are only A students and just below only B students.\n", "\n", "**Answer:** the density test passes while covariate balance fails. $\\blacksquare$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Very low birth weight and heaping\n", "\n", "A study finds that babies just below 1,500 g (who get extra care) have lower mortality than those just above. Re-analysis shows a spike of babies recorded at exactly 1,500 g, with high mortality, and that dropping them halves the estimated effect. What do you conclude?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Heaped babies at exactly 1,500 g fall on the \"above\" side and have high mortality for reasons unrelated to care (they come from hospitals with poorer measurement and care).\n", "\n", "**Step 2.** They inflate the right limit, exaggerating the benefit of extra care.\n", "\n", "**Answer:** the donut estimate is more credible; the original effect was partly an artefact of heaping.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Exam scores just at the pass mark\n", "\n", "On a state exam with a pass mark of 65, 900 students score 63–64 and 2,100 score 65–66 (equal-width bins, otherwise smooth distribution). Interpret, and say what this means for an RD of passing on later outcomes.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\ln(2{,}100/900) \\approx 0.85$, SE $\\approx \\sqrt{1/900 + 1/2{,}100} \\approx 0.040$: $z \\approx 21$.\n", "\n", "**Step 2.** Graders bumped students just below 65 up to the pass mark.\n", "\n", "**Answer:** the RD at 65 is invalid as is: students just above include bumped-up students who differ from others. Possible fixes: donut RD, or using an instrument based on the unmanipulated score.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Close elections\n", "\n", "A study of close US House races finds that the party holding the seat wins close elections more often than chance, and that bare winners have more campaign money. What does this do to the incumbency RD?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Pre-determined covariates (prior incumbency, money) jump at the 0 margin: covariate balance fails.\n", "\n", "**Step 2.** Some precise control over close outcomes (recounts, resources, local knowledge) breaks the as-good-as-random comparison.\n", "\n", "**Answer:** the incumbency effect may be overstated; studies respond with covariate adjustment, other electoral contexts or narrower windows, and debate persists.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Sorting in a simulation\n", "\n", "In the simulation below, 60% of units just below the cutoff with above-average untreated outcomes move just above it. Compare the RD estimate, the density test and a donut estimate with the true effect of 3.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The density test detects excess mass above the cutoff (log ratio about 0.64, $z > 10$).\n", "\n", "**Step 2.** The RD estimate is biased upwards (about 3.65) because movers have higher untreated outcomes (Exercise 8).\n", "\n", "**Step 3.** Dropping a 0.05-wide donut around the cutoff brings the estimate back near 3 (about 2.8), with more noise.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · A validity checklist\n", "\n", "Write a checklist of validity checks for an RD estimating the effect of a means-tested child benefit (income cutoff) on child health.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Institutional knowledge: can families adjust reported income to qualify? Does anything else switch at the cutoff?\n", "\n", "**Step 2.** Density test of reported income at the cutoff.\n", "\n", "**Step 3.** Covariate balance: parental education, household size, child birth weight.\n", "\n", "**Step 4.** Placebo cutoffs and outcomes measured before the benefit (baseline health).\n", "\n", "**Step 5.** Donut RD and bandwidth sensitivity; RBC intervals.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "z = lambda n1, n2: np.log(n2 / n1) / np.sqrt(1 / n1 + 1 / n2)\n", "print(f\"Ex 1: log ratio {np.log(620 / 480):.3f}, z {z(480, 620):.1f} | Ex 12: log ratio {np.log(2100 / 900):.2f}, z {z(900, 2100):.1f}\")\n", "\n", "rng = np.random.default_rng(77); n = 40_000\n", "x = rng.uniform(-1, 1, n); y0 = 10 + 4 * x + rng.normal(0, 2, n)\n", "mover = (x < 0) & (x > -.05) & (y0 - 10 - 4 * x > 0) & (rng.random(n) < .6)\n", "xm = x.copy(); xm[mover] = rng.uniform(0, .05, mover.sum())\n", "y = y0 + 3 * (xm >= 0)\n", "def rd(x, y, h=.3):\n", " def side(xx, yy):\n", " w = np.clip(1 - np.abs(xx) / h, 0, None); m = w > 0\n", " X = np.column_stack([np.ones(m.sum()), xx[m]]); W = w[m]\n", " return np.linalg.solve(X.T @ (W[:, None] * X), X.T @ (W * yy[m]))[0]\n", " r = x >= 0\n", " return side(x[r], y[r]) - side(-x[~r], y[~r])\n", "nl, nr = ((xm < 0) & (xm >= -.05)).sum(), ((xm >= 0) & (xm < .05)).sum()\n", "print(f\"movers {mover.sum()}; density bins {nl} vs {nr}: log ratio {np.log(nr / nl):.2f}, z {z(nl, nr):.1f}\")\n", "keep = np.abs(xm) > .05\n", "print(f\"RD with sorting {rd(xm, y):.2f} donut {rd(xm[keep], y[keep]):.2f} no sorting {rd(x, y0 + 3 * (x >= 0)):.2f} (truth 3)\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- RD is credible when units cannot precisely control which side of the cutoff they land on.\n", "- Check the density of the running variable, balance of pre-determined covariates, placebo cutoffs and outcomes, and robustness to a donut.\n", "- Sorting by units with different potential outcomes biases the jump; two-sided manipulation can hide from density tests but not from covariate checks." ] } ] }