{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 75: Bandwidth Selection & Bias Correction\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "The bandwidth trades bias against variance. The MSE-optimal bandwidth balances them so well that the bias is no longer negligible relative to the standard error, and conventional confidence intervals under-cover. Robust bias-corrected inference fixes this by subtracting an estimate of the bias and accounting for its uncertainty.\n", "\n", "**Learning outcomes**\n", "\n", "- Derive the MSE-optimal bandwidth and its n^(−1/5) rate.\n", "- Show that at the MSE-optimal bandwidth the bias is half the standard error, and compute the resulting coverage loss.\n", "- Explain robust bias-corrected (RBC) confidence intervals and why conventional standard errors are too small after bias correction.\n", "- Design a bandwidth robustness analysis for an RD study." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The trade-off\n", "\n", "For a local linear RD estimator with bandwidth $h$ and sample size $n$,\n", "\n", "$$\n", "\\text{Bias} \\approx Bh^2, \\qquad \\operatorname{Var} \\approx \\frac{V}{nh},\n", "$$\n", "\n", "where $B$ depends on the curvature of $E[Y \\mid X]$ near the cutoff and $V$ on the outcome variance and the density of $X$ at $c$.\n", "\n", "### MSE-optimal bandwidth\n", "\n", "Minimising $\\text{MSE}(h) = B^2h^4 + V/(nh)$ gives\n", "\n", "$$\n", "h_{MSE} = \\Big(\\frac{V}{4B^2n}\\Big)^{1/5} \\propto n^{-1/5}.\n", "$$\n", "\n", "Data-driven selectors (Imbens–Kalyanaraman; Calonico–Cattaneo–Titiunik, CCT) estimate $B$ (using a pilot local quadratic fit) and $V$, then plug in. Different bandwidths can be used on each side.\n", "\n", "### The inference problem\n", "\n", "At $h_{MSE}$, squared bias equals one quarter of the variance, so\n", "\n", "$$\n", "\\frac{\\text{Bias}}{\\operatorname{SE}} = \\frac12.\n", "$$\n", "\n", "A conventional 95% interval $\\hat\\tau \\pm 1.96\\,\\operatorname{SE}$ then covers only about 92% of the time, and less when bias is underestimated.\n", "\n", "### Fixes\n", "\n", "- **Undersmoothing:** use a smaller bandwidth than $h_{MSE}$ so bias is negligible; the point estimate becomes noisier and the right amount of undersmoothing is unclear.\n", "- **Robust bias correction (Calonico, Cattaneo and Titiunik, 2014):** estimate the bias with a local quadratic fit (pilot bandwidth $b$), subtract it, and compute a standard error that includes the variability of the bias estimate. With $b = h$, the bias-corrected estimator coincides with the local quadratic estimator, and RBC inference is local quadratic inference at bandwidth $h$.\n", "\n", "**Current practice:** report the local linear estimate at $h_{MSE}$ with the RBC confidence interval; show sensitivity to halving and doubling the bandwidth. Coverage-error-optimal bandwidths (smaller than $h_{MSE}$) are an alternative for inference.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · An optimal bandwidth\n", "\n", "With $B = 2$, $V = 4$ and $n = 1{,}000$, compute $h_{MSE}$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $h = (4/(4 \\times 4 \\times 1{,}000))^{1/5} = (1/4{,}000)^{1/5}$.\n", "\n", "**Step 2.** $\\ln 4{,}000 \\approx 8.29$; divided by 5: 1.66; $e^{-1.66} \\approx 0.19$.\n", "\n", "**Answer:** about 0.19 units of the running variable.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · More data\n", "\n", "If the sample grows 32-fold, how does $h_{MSE}$ change?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $h \\propto n^{-1/5}$; $32^{-1/5} = 1/2$.\n", "\n", "**Answer:** it halves.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Coverage at the optimum\n", "\n", "With bias equal to half the SE, what is the coverage of a nominal 95% interval?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The t-statistic is $N(0.5, 1)$ around the truth.\n", "\n", "**Step 2.** Coverage $= P(|Z + 0.5| < 1.96) = \\Phi(1.46) - \\Phi(-2.46) \\approx 0.928 - 0.007 = 0.921$.\n", "\n", "**Answer:** about 92%.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Undersmoothing\n", "\n", "To make the bias at most 10% of the SE, by what factor must the bandwidth shrink relative to $h_{MSE}$?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Bias/SE $\\propto h^2 \\cdot h^{1/2} = h^{5/2}$.\n", "\n", "**Step 2.** We need the ratio to fall from 0.5 to 0.1: factor 5, so $h$ shrinks by $5^{2/5} \\approx 1.90$.\n", "\n", "**Answer:** use roughly half the MSE-optimal bandwidth; the SE then rises by $\\sqrt{1.9} \\approx 1.38$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Ingredients of RBC\n", "\n", "List what is estimated in robust bias-corrected inference.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The local linear estimate $\\hat\\tau_h$ with bandwidth $h$.\n", "\n", "**Step 2.** The bias $\\hat B$ from a local quadratic fit with pilot bandwidth $b$.\n", "\n", "**Step 3.** A standard error for $\\hat\\tau_h - \\hat B$ that includes the variance of $\\hat B$ and its covariance with $\\hat\\tau_h$.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Deriving the optimal bandwidth\n", "\n", "Minimise $B^2h^4 + V/(nh)$ over $h > 0$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Derivative: $4B^2h^3 - V/(nh^2) = 0$.\n", "\n", "**Step 2.** $h^5 = V/(4B^2n)$, so $h = (V/(4B^2n))^{1/5}$.\n", "\n", "**Step 3.** The second derivative $12B^2h^2 + 2V/(nh^3) > 0$: a minimum. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Bias is half the SE\n", "\n", "Show that at $h_{MSE}$, $\\text{Bias}^2 = \\operatorname{Var}/4$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** From Exercise 6, $4B^2h^5 = V/n$, so $B^2h^4 = V/(4nh)$.\n", "\n", "**Step 2.** The left side is $\\text{Bias}^2$; the right side is $\\operatorname{Var}/4$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · The convergence rate\n", "\n", "Show that at $h_{MSE}$ the root mean squared error is of order $n^{-2/5}$, slower than the parametric $n^{-1/2}$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\text{MSE} = B^2h^4 + V/(nh)$ with $h \\propto n^{-1/5}$.\n", "\n", "**Step 2.** $B^2h^4 \\propto n^{-4/5}$ and $V/(nh) \\propto n^{-1 + 1/5} = n^{-4/5}$.\n", "\n", "**Step 3.** $\\text{RMSE} \\propto n^{-2/5}$. $\\blacksquare$ RD uses only data near the cutoff, so it learns more slowly than an experiment.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Why bias correction needs a new SE\n", "\n", "Let $\\hat\\tau_{bc} = \\hat\\tau - \\hat B$. Show $\\operatorname{Var}(\\hat\\tau_{bc}) = \\operatorname{Var}(\\hat\\tau) + \\operatorname{Var}(\\hat B) - 2\\operatorname{Cov}(\\hat\\tau, \\hat B)$ and explain why using $\\operatorname{SE}(\\hat\\tau)$ alone is wrong.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Variance of a difference.\n", "\n", "**Step 2.** $\\hat B$ is estimated from the same (or overlapping) data with a curvature estimate, which is noisy: $\\operatorname{Var}(\\hat B)$ is not negligible in large samples when $b$ is of the same order as $h$.\n", "\n", "**Answer:** ignoring it understates uncertainty; RBC uses the full variance. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Choosing the bandwidth by eye\n", "\n", "Explain why picking the bandwidth that gives the most significant estimate invalidates inference.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Estimates at different bandwidths are noisy, and some will be significant by chance.\n", "\n", "**Step 2.** Choosing the most significant inflates the probability of rejecting a true null above 5% (specification searching).\n", "\n", "**Answer:** pre-specify a data-driven bandwidth and report sensitivity across bandwidths, not the best one. $\\blacksquare$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · A scholarship RD bandwidth\n", "\n", "A pilot local quadratic fit suggests $B = 0.5$ (enrolment probability per squared score unit, in standardised score units), the variance constant is $V = 2$, and there are $n = 5{,}000$ students. Compute $h_{MSE}$, and the bandwidth after a 50% undersmoothing.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $h = (2/(4 \\times 0.25 \\times 5{,}000))^{1/5} = (2/5{,}000)^{1/5} = (0.0004)^{0.2}$.\n", "\n", "**Step 2.** $\\ln 0.0004 \\approx -7.82$; $/5 = -1.56$; $e^{-1.56} \\approx 0.21$.\n", "\n", "**Answer:** $h_{MSE} \\approx 0.21$ sd of the score; undersmoothed about 0.10.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Coverage in a simulation\n", "\n", "In the simulation below (true jump 5, curved regression function, 1,000 observations), compare the coverage of conventional local linear intervals with local quadratic (RBC with $b = h$) intervals at bandwidths 0.15, 0.3 and 0.5.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Small $h$: both cover about 92–93%; the local quadratic interval is wider.\n", "\n", "**Step 2.** At $h = 0.5$, the local linear estimate is biased (about +0.5) and its interval covers only about 86%.\n", "\n", "**Step 3.** The local quadratic interval keeps coverage near 94% at every bandwidth.\n", "\n", "**Answer:** robust bias correction protects inference when the bandwidth is large enough for bias to matter.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · A bandwidth robustness table\n", "\n", "For an RD on the effect of a pension eligibility age on retirement, outline a robustness table.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Rows: $h_{MSE}$, $h_{MSE}/2$, $2h_{MSE}$, and the coverage-error-optimal bandwidth.\n", "\n", "**Step 2.** Columns: point estimate, RBC 95% CI, number of observations on each side.\n", "\n", "**Step 3.** Also: local quadratic fits, triangular versus uniform kernel.\n", "\n", "**Answer:** stable estimates across rows support the result; strong sensitivity signals curvature or too few observations near the cutoff.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Different bandwidths on each side\n", "\n", "Mortality data around a very-low-birth-weight cutoff have many more observations above than below the threshold, and curvature differs. Why might separate bandwidths help?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Each side's MSE depends on its own density, variance and curvature.\n", "\n", "**Step 2.** Separate bandwidths use more data where it is plentiful and the function is flat, less where it curves.\n", "\n", "**Answer:** lower overall MSE for the jump estimate; CCT-style selectors provide side-specific bandwidths.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Few observations near the cutoff\n", "\n", "An RD on a school-entry age cutoff has only 60 children within a month of the cutoff. What alternative to continuity-based inference is appropriate?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Asymptotic local polynomial inference may be unreliable with so few observations.\n", "\n", "**Step 2.** Local randomisation: treat children born within a narrow window as if randomly assigned, check covariate balance, and use Fisher randomisation inference (Day 37).\n", "\n", "**Answer:** a local-randomisation analysis in a small window, reported alongside the continuity-based estimate.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from scipy.stats import norm\n", "\n", "print(\"Ex 1 h_MSE:\", round((4 / (4 * 4 * 1000)) ** .2, 3), \" Ex 11:\", round((2 / (4 * .25 * 5000)) ** .2, 3),\n", " \" Ex 3 coverage:\", round(norm.cdf(1.96 - .5) - norm.cdf(-1.96 - .5), 3), \" Ex 4 factor:\", round(5 ** .4, 2))\n", "\n", "rng = np.random.default_rng(75)\n", "mu0 = lambda x: 50 + 8 * x + 30 * x ** 2 - 20 * x ** 3\n", "def side(x, y, h, p):\n", " w = np.clip(1 - np.abs(x) / h, 0, None); m = w > 0\n", " X = np.column_stack([x[m] ** k for k in range(p + 1)]); W = w[m]\n", " A = np.linalg.inv(X.T @ (W[:, None] * X)); b = A @ (X.T @ (W * y[m])); r = y[m] - X @ b\n", " V = A @ (X.T @ ((W ** 2 * r ** 2)[:, None] * X)) @ A # heteroskedasticity-robust\n", " return b[0], V[0, 0]\n", "def rd(x, y, h, p):\n", " r = x >= 0\n", " a, va = side(x[r], y[r], h, p); b, vb = side(-x[~r], y[~r], h, p)\n", " return a - b, np.sqrt(va + vb)\n", "print(\" h LL bias LL sd LL cover | LQ bias LQ sd LQ cover\")\n", "for h in (.15, .3, .5):\n", " out = {1: [], 2: []}; cov = {1: 0, 2: 0}\n", " for _ in range(1000):\n", " x = rng.uniform(-1, 1, 1000); y = mu0(x) + 5 * (x >= 0) + rng.normal(0, 3, 1000)\n", " for p in (1, 2):\n", " t, s = rd(x, y, h, p); out[p].append(t); cov[p] += abs(t - 5) < 1.96 * s\n", " print(f\"{h:4.2f} {np.mean(out[1]) - 5:7.2f} {np.std(out[1]):5.2f} {cov[1] / 1000:7.3f} | \"\n", " f\"{np.mean(out[2]) - 5:7.2f} {np.std(out[2]):5.2f} {cov[2] / 1000:7.3f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- The MSE-optimal bandwidth shrinks like n^(−1/5) and leaves a bias equal to half the standard error.\n", "- Conventional intervals at that bandwidth under-cover; robust bias-corrected intervals subtract the estimated bias and account for its variance.\n", "- Pre-specify a data-driven bandwidth, report RBC intervals, and show sensitivity to the bandwidth." ] } ] }