{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 65: Weak Instruments & Robust Inference\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "When the instrument barely moves the treatment, 2SLS is biased towards OLS, its distribution is far from normal and the usual confidence intervals under-cover. The first-stage F statistic diagnoses the problem; the Anderson–Rubin test gives inference that stays valid however weak the instrument.\n", "\n", "**Learning outcomes**\n", "\n", "- Compute and interpret the first-stage F statistic and the concentration parameter.\n", "- Explain and quantify the bias of 2SLS towards OLS with weak instruments.\n", "- Carry out an Anderson–Rubin test and invert it into a confidence set, which may be unbounded.\n", "- Apply current reporting practice (F > 10 is not enough for t-tests) in empirical work." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### First-stage strength\n", "\n", "In the first stage $X = \\pi^\\top\\mathbf Z + v$ with $l$ excluded instruments, strength is measured by the **concentration parameter** $\\mu^2 = \\pi^\\top\\mathbf Z^\\top\\mathbf Z\\pi/\\sigma_v^2$, estimated through the **first-stage F statistic** for $H_0: \\pi = 0$. Roughly $E[F] \\approx 1 + \\mu^2/l$. With one instrument, $F = t^2$ for the first-stage coefficient.\n", "\n", "### Bias towards OLS\n", "\n", "With weak instruments, 2SLS is biased in the direction of OLS. A useful approximation (many instruments, homoskedastic errors):\n", "\n", "$$\n", "\\frac{E[\\hat\\beta_{2SLS}] - \\beta}{\\operatorname{plim}\\hat\\beta_{OLS} - \\beta} \\approx \\frac{1}{F}.\n", "$$\n", "\n", "When the instruments are irrelevant ($\\pi = 0$), 2SLS is centred on the OLS probability limit: weak IV \"finds\" the OLS answer. Bound, Jaeger and Baker (1995) reproduced quarter-of-birth results with randomly generated instruments.\n", "\n", "### Bad inference\n", "\n", "The 2SLS t-statistic is not approximately normal when $F$ is small: Wald confidence intervals ($\\hat\\beta \\pm 1.96\\,\\operatorname{SE}$) cover less than 95%, sometimes far less when endogeneity is strong. The rule of thumb $F > 10$ (Staiger and Stock; Stock and Yogo) was designed to keep bias below about 10% of OLS bias. For a just-identified model, Lee, McCrary, Moreira and Porter (2022) show the usual t-test has correct 5% size only if $F > 104.7$; for smaller $F$ the critical value must be raised (their tF procedure).\n", "\n", "### The Anderson–Rubin test\n", "\n", "To test $H_0: \\beta = \\beta_0$, regress $Y - \\beta_0X$ on the instruments (and controls) and test that all instrument coefficients are zero. Under $H_0$, $Y - \\beta_0X = \\varepsilon$, which is unrelated to $\\mathbf Z$, so the test has correct size **whatever the first-stage strength**. Collecting all $\\beta_0$ not rejected gives an AR **confidence set**, which can be:\n", "\n", "- a bounded interval (strong enough first stage);\n", "- the union of two rays, or the whole real line (weak first stage): the data are honestly uninformative.\n", "\n", "### Practice\n", "\n", "Report the first stage and its F (robust or effective F with heteroskedastic or clustered errors), the reduced form (an unbiased intention-to-treat estimate whatever the first stage), and AR or tF confidence sets when $F$ is not large.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · F from t\n", "\n", "A first-stage coefficient has t-statistic 3. What is the first-stage F with one instrument? Is the instrument \"strong\" by the F > 10 rule?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $F = t^2 = 9$.\n", "\n", "**Answer:** 9, just below 10: weak by the rule of thumb, and far below the 104.7 needed for standard t-tests.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Bias towards OLS\n", "\n", "OLS overstates a return by 0.05. With first-stage F = 5, roughly how biased is 2SLS?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Relative bias $\\approx 1/F = 0.2$.\n", "\n", "**Step 2.** 2SLS bias $\\approx 0.2 \\times 0.05 = 0.01$, in the same direction as OLS.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · An Anderson–Rubin test\n", "\n", "Reduced form 0.06, first stage 0.4. For each $\\beta_0$, the coefficient on $Z$ when regressing $Y - \\beta_0X$ on $Z$ is $0.06 - 0.4\\beta_0$; take its standard error as 0.02 near the estimate. Is $\\beta_0 = 0$ rejected? Find the AR 95% interval.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\beta_0 = 0$: coefficient 0.06, t = 3: rejected.\n", "\n", "**Step 2.** Not rejected when $|0.06 - 0.4\\beta_0| \\le 1.96 \\times 0.02 = 0.0392$.\n", "\n", "**Step 3.** $0.0208 \\le 0.4\\beta_0 \\le 0.0992$, so $\\beta_0 \\in [0.052, 0.248]$.\n", "\n", "**Answer:** reject zero; AR interval [0.052, 0.248] around the IV estimate 0.15.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Which statistic to report?\n", "\n", "A paper with one instrument reports first-stage F = 25 and a 2SLS 95% interval of [0.02, 0.30]. What would you ask for?\n", "\n", "
Show solution\n", "\n", "**Step 1.** F = 25 passes the F > 10 rule but is below 104.7, so the t-based interval may under-cover.\n", "\n", "**Answer:** an AR confidence set (or tF-adjusted interval) and the reduced form with its own standard error.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Irrelevant instruments\n", "\n", "What does 2SLS estimate if the instrument is completely irrelevant ($\\pi = 0$)?\n", "\n", "
Show solution\n", "\n", "**Step 1.** There is no exogenous variation in $X$ to use.\n", "\n", "**Answer:** 2SLS does not converge to $\\beta$; its distribution is centred near the OLS probability limit and does not shrink with $n$.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · AR has correct size\n", "\n", "Assume $Y = \\beta X + \\varepsilon$ with $\\varepsilon \\sim N(0, \\sigma^2)$ independent of $Z$ (fixed). Prove that under $H_0: \\beta = \\beta_0$ the AR statistic has an exact F distribution whatever $\\pi$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Under $H_0$, $Y - \\beta_0X = \\varepsilon$.\n", "\n", "**Step 2.** Regressing $\\varepsilon$ on $Z$ and testing the coefficients is a classical F test with normal errors independent of the regressors.\n", "\n", "**Step 3.** Its null distribution is $F_{l, n - l - 1}$, which does not involve $X$ or $\\pi$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Irrelevant instruments centre on OLS\n", "\n", "Scalar case, no constant: $X = v$ ($\\pi = 0$), $Y = \\beta X + \\varepsilon$, with $(\\varepsilon, v)$ jointly normal, $\\operatorname{Cov}(\\varepsilon, v) = \\sigma_{\\varepsilon v}$, and $Z$ independent of both. Show that $\\hat\\beta_{IV} - \\beta = \\frac{\\mathbf z^\\top\\boldsymbol\\varepsilon}{\\mathbf z^\\top\\mathbf v}$ and that, conditional on $\\mathbf z^\\top\\mathbf v$, its mean is $\\sigma_{\\varepsilon v}/\\sigma_v^2$, the OLS bias.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\hat\\beta_{IV} = \\mathbf z^\\top\\mathbf y/\\mathbf z^\\top\\mathbf x = \\beta + \\mathbf z^\\top\\boldsymbol\\varepsilon/\\mathbf z^\\top\\mathbf v$.\n", "\n", "**Step 2.** Decompose $\\varepsilon = (\\sigma_{\\varepsilon v}/\\sigma_v^2)v + w$, with $w$ independent of $v$ and $Z$.\n", "\n", "**Step 3.** $\\mathbf z^\\top\\boldsymbol\\varepsilon/\\mathbf z^\\top\\mathbf v = \\sigma_{\\varepsilon v}/\\sigma_v^2 + \\mathbf z^\\top\\mathbf w/\\mathbf z^\\top\\mathbf v$.\n", "\n", "**Step 4.** Given $\\mathbf z^\\top\\mathbf v$ (and $\\mathbf z$), $\\mathbf z^\\top\\mathbf w$ has mean 0. $\\blacksquare$ So the estimator is centred at $\\beta + \\sigma_{\\varepsilon v}/\\sigma_v^2 = \\operatorname{plim}\\hat\\beta_{OLS}$, with heavy tails from the random denominator.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · When AR sets are unbounded\n", "\n", "Just-identified case: let the reduced form be $\\hat\\rho$, the first stage $\\hat\\pi$, with variances $s_\\rho^2, s_\\pi^2$ and covariance 0. The AR set is $\\{\\beta: (\\hat\\rho - \\beta\\hat\\pi)^2 \\le c\\,(s_\\rho^2 + \\beta^2 s_\\pi^2)\\}$ with $c = 1.96^2$. Show that it is unbounded when $\\hat\\pi^2 < c\\,s_\\pi^2$, i.e. when the first stage is not significant at 5%.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Rearrange: $(\\hat\\pi^2 - c\\,s_\\pi^2)\\beta^2 - 2\\hat\\rho\\hat\\pi\\beta + (\\hat\\rho^2 - c\\,s_\\rho^2) \\le 0$.\n", "\n", "**Step 2.** If the leading coefficient $\\hat\\pi^2 - c\\,s_\\pi^2 < 0$, the quadratic tends to $-\\infty$ as $|\\beta| \\to \\infty$, so all large $|\\beta|$ satisfy the inequality.\n", "\n", "**Step 3.** The set therefore contains both tails: unbounded (two rays, or the whole line). $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · F equals t squared\n", "\n", "With one instrument, prove that the first-stage F statistic equals the square of the first-stage t statistic.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The F test of one restriction is $F = (\\text{restricted SSR} - \\text{unrestricted SSR})/(\\text{unrestricted SSR}/(n - k))$.\n", "\n", "**Step 2.** For a single coefficient, the numerator equals $\\hat\\pi^2\\sum\\tilde z_i^2$ (FWL), and $\\hat\\sigma_v^2 = \\text{SSR}/(n - k)$.\n", "\n", "**Step 3.** So $F = \\hat\\pi^2\\sum\\tilde z_i^2/\\hat\\sigma_v^2 = \\big(\\hat\\pi/\\operatorname{SE}(\\hat\\pi)\\big)^2 = t^2$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · The reduced form is always informative\n", "\n", "Explain why the reduced form $\\hat\\rho$ is a valid estimate of a causal effect (of the instrument) whatever the first-stage strength, and how its sign relates to that of $\\beta$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $Z$ is as good as random, so the regression of $Y$ on $Z$ estimates the causal effect of $Z$ (an ITT), with standard inference.\n", "\n", "**Step 2.** Under exclusion and a constant effect, $\\rho = \\beta\\pi$; if the sign of $\\pi$ is known, the sign of $\\rho$ is the sign of $\\beta$.\n", "\n", "**Answer:** testing $\\rho = 0$ is a weak-instrument-robust test of $\\beta = 0$ (it is the AR test at $\\beta_0 = 0$). $\\blacksquare$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Quarter of birth with many instruments\n", "\n", "Using 180 quarter-of-birth × state × year interactions as instruments, a study of returns to schooling gets a first-stage F of 2 and a 2SLS estimate of 0.061; OLS is 0.063. With 3 quarter dummies, F = 30 and 2SLS is 0.081. Which estimate is more credible?\n", "\n", "
Show solution\n", "\n", "**Step 1.** With F = 2, the relative bias towards OLS is about 50%: the 180-instrument estimate is pulled almost to OLS.\n", "\n", "**Step 2.** The 3-instrument estimate has a stronger first stage, and less bias.\n", "\n", "**Answer:** the 3-instrument estimate (0.081); many weak instruments manufacture an estimate close to OLS.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Coverage in a simulation\n", "\n", "In the Monte Carlo below (one instrument, strong endogeneity, true $\\beta = 1$), what happens to the median 2SLS estimate and to the coverage of the Wald and AR intervals as the first-stage F rises from about 1 to about 46?\n", "\n", "
Show solution\n", "\n", "**Step 1.** F ≈ 1: the median 2SLS estimate is near the OLS limit (1.8); Wald coverage about 88%; AR 95%.\n", "\n", "**Step 2.** F ≈ 11: median close to 1, but Wald coverage still only about 93%.\n", "\n", "**Step 3.** F ≈ 46: Wald coverage about 95%; AR stays at 95% throughout.\n", "\n", "**Answer:** F > 10 removes most bias but not all over-rejection; AR is valid everywhere.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Distance to a specialty hospital\n", "\n", "Distance to a cardiac-care hospital instruments for receiving catheterisation (an invasive heart procedure) in a study of survival. The first-stage t is 1.5. What will the AR confidence set look like, and what can still be reported?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $t^2 = 2.25 < 3.84$: the first stage is not significant at 5%.\n", "\n", "**Step 2.** By Exercise 8, the AR set is unbounded.\n", "\n", "**Step 3.** The reduced form (effect of distance on survival) can still be reported with standard inference.\n", "\n", "**Answer:** the data cannot bound the effect of catheterisation; say so, report the reduced form, and look for a stronger instrument.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Judge leniency designs\n", "\n", "Defendants are randomly assigned to judges who differ in leniency; leniency instruments for incarceration. With 200 judges, the first stage uses 200 dummies. What weak-instrument problem arises, and what is the usual fix?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Many instruments, each estimated from few cases: 2SLS is biased towards OLS (many-instrument bias).\n", "\n", "**Step 2.** Fix: construct a single leave-one-out leniency measure for each judge (the average incarceration rate of the judge's other cases) and use it as one instrument.\n", "\n", "**Answer:** one strong instrument (high F) instead of 200 weak ones; the leave-one-out construction avoids the mechanical correlation with the defendant's own outcome.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · A reporting checklist\n", "\n", "Write a weak-instrument reporting checklist for an IV paper on the effect of microcredit on business profits instrumented by branch openings.\n", "\n", "
Show solution\n", "\n", "**Step 1.** First stage: coefficient, robust or effective F, and a plot of branch openings against credit take-up.\n", "\n", "**Step 2.** Reduced form: effect of branch openings on profits with its confidence interval.\n", "\n", "**Step 3.** 2SLS estimate with AR (or tF) confidence set, and a note on whether it is bounded.\n", "\n", "**Step 4.** Comparison with OLS and, if over-identified, with single-instrument estimates (Day 66).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(65); n = 500\n", "print(\" pi mean F median 2SLS Wald coverage AR coverage\")\n", "for pi in (0.0, 0.07, 0.14, 0.30):\n", " F, est, cw, ca = [], [], 0, 0\n", " for _ in range(2000):\n", " z = rng.normal(size=n); e = rng.normal(size=n); v = .8 * e + .6 * rng.normal(size=n)\n", " x = pi * z + v; y = 1.0 * x + e # true beta = 1, OLS limit 1.8\n", " z, x, y = z - z.mean(), x - x.mean(), y - y.mean()\n", " g = z @ x / (z @ z); r = x - g * z\n", " F.append(g**2 * (z @ z) / (r @ r / (n - 2)))\n", " b = z @ y / (z @ x); u = y - b * x\n", " se = np.sqrt(u @ u / (n - 2) * (z @ z)) / abs(z @ x)\n", " cw += abs(b - 1) / se < 1.96; est.append(b)\n", " w = y - 1.0 * x; h = z @ w / (z @ z); rr = w - h * z # Anderson-Rubin at beta0 = 1\n", " ca += abs(h / np.sqrt(rr @ rr / (n - 2) / (z @ z))) < 1.96\n", " print(f\"{pi:.2f} {np.mean(F):6.1f} {np.median(est):10.3f} {cw / 2000:12.3f} {ca / 2000:10.3f}\")\n", "\n", "# Exercise 3: AR interval\n", "print(\"AR interval:\", np.round([(0.06 - 1.96 * .02) / .4, (0.06 + 1.96 * .02) / .4], 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Weak instruments bias 2SLS towards OLS (roughly by a factor 1/F) and distort t-based inference.\n", "- Report the first-stage F, the reduced form, and Anderson–Rubin (or tF) confidence sets; F > 10 limits bias but does not make t-tests reliable.\n", "- AR sets are valid for any strength and become unbounded when the first stage is not significant: an honest \"we don't know\"." ] } ] }