{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 66: Over-Identification & Instrument Validity\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "With more instruments than endogenous regressors, the instruments can be checked against each other. The Sargan–Hansen J test, endogeneity (Hausman and control-function) tests, and sensitivity analysis for \"plausibly exogenous\" instruments, with a clear view of what each can and cannot reveal.\n", "\n", "**Learning outcomes**\n", "\n", "- Compute the Sargan statistic as n R² and state its degrees of freedom.\n", "- Test whether the regressor is endogenous with the control-function (Durbin–Wu–Hausman) approach.\n", "- Prove that over-identification tests have no power against instruments that are invalid in the same direction.\n", "- Bound an IV estimate when the exclusion restriction may be slightly violated." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Over-identifying restrictions\n", "\n", "With $l$ instruments and $k$ endogenous regressors ($l > k$), the moment conditions $E[\\mathbf z_i\\varepsilon_i] = \\mathbf 0$ are more than needed. If all instruments are valid, every just-identified subset estimates the same $\\beta$ (with a constant effect), and the residual should be unrelated to every instrument.\n", "\n", "### The Sargan–Hansen J test\n", "\n", "1. Estimate by 2SLS and compute residuals $\\hat\\varepsilon = \\mathbf y - \\mathbf X\\hat{\\boldsymbol\\beta}$.\n", "2. Regress $\\hat\\varepsilon$ on all instruments (and controls); record $R^2$.\n", "3. $J = nR^2 \\sim \\chi^2_{l - k}$ under the null that all instruments are valid (homoskedastic errors). Hansen's J, the minimised two-step GMM objective, is the robust version.\n", "\n", "### What J can and cannot do\n", "\n", "- If one instrument is invalid and others valid, J has power to detect the disagreement, but it cannot say which instrument is wrong.\n", "- If all instruments are invalid **in the same way** (estimates biased by the same amount), J has no power (Exercise 9).\n", "- With heterogeneous effects, valid instruments can identify different LATEs (Day 63), so J can reject even when every instrument is valid.\n", "\n", "### Is the regressor endogenous?\n", "\n", "The **Durbin–Wu–Hausman** test compares OLS and IV. A convenient regression version is the **control function**: (1) run the first stage and save the residual $\\hat v$; (2) regress $Y$ on $X$, controls and $\\hat v$. The coefficient on $X$ equals 2SLS, and a significant coefficient on $\\hat v$ rejects exogeneity of $X$.\n", "\n", "### Plausibly exogenous instruments\n", "\n", "If $Z$ has a small direct effect $\\gamma$ on $Y$, then $\\beta_{IV} = \\beta + \\gamma/\\pi$. Allowing $\\gamma \\in [-\\delta, \\delta]$, the union of the IV confidence intervals shifted by $\\gamma/\\pi$ gives a sensitivity interval (Conley, Hansen and Rossi). Weak first stages magnify small violations.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Degrees of freedom\n", "\n", "A model has 2 endogenous regressors and 5 excluded instruments. What are the degrees of freedom of the J test?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $l - k = 5 - 2 = 3$ over-identifying restrictions.\n", "\n", "**Answer:** $\\chi^2_3$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Computing J\n", "\n", "With $n = 1{,}000$, regressing the 2SLS residuals on 3 instruments (1 endogenous regressor) gives $R^2 = 0.006$. Compute J and test at 5% ($\\chi^2_{2, 0.95} = 5.99$).\n", "\n", "
Show solution\n", "\n", "**Step 1.** $J = 1{,}000 \\times 0.006 = 6.0$.\n", "\n", "**Step 2.** $6.0 > 5.99$: reject (barely).\n", "\n", "**Answer:** the instruments disagree more than chance would explain, at the 5% level.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Comparing two just-identified estimates\n", "\n", "Two instruments give 0.15 (SE 0.02) and 0.10 (SE 0.04), with approximately independent estimates. Test their equality.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Difference: 0.05; SE $\\sqrt{0.02^2 + 0.04^2} = \\sqrt{0.002} \\approx 0.0447$.\n", "\n", "**Step 2.** $z = 0.05/0.0447 \\approx 1.12 < 1.96$.\n", "\n", "**Answer:** not rejected; the instruments are statistically compatible.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · A control-function test\n", "\n", "In the regression of $Y$ on $X$ and the first-stage residual $\\hat v$, the coefficient on $\\hat v$ has $t = 2.5$. What do you conclude?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $|t| > 1.96$: reject $H_0$ that $X$ is exogenous.\n", "\n", "**Answer:** OLS and IV differ significantly; use IV (assuming the instrument is valid).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Just identified\n", "\n", "What is J in a just-identified model, and why?\n", "\n", "
Show solution\n", "\n", "**Step 1.** With $l = k$, 2SLS sets $\\mathbf Z^\\top\\hat{\\boldsymbol\\varepsilon} = \\mathbf 0$ exactly (Day 64, Exercise 7).\n", "\n", "**Step 2.** The residuals are orthogonal to all instruments, so $R^2 = 0$ and $J = 0$.\n", "\n", "**Answer:** zero: just-identified models have nothing to test.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · J = 0 at an exact fit\n", "\n", "Prove that in a just-identified model $\\mathbf Z^\\top(\\mathbf y - \\mathbf X\\hat{\\boldsymbol\\beta}_{IV}) = \\mathbf 0$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\hat{\\boldsymbol\\beta}_{IV} = (\\mathbf Z^\\top\\mathbf X)^{-1}\\mathbf Z^\\top\\mathbf y$.\n", "\n", "**Step 2.** $\\mathbf Z^\\top\\mathbf X\\hat{\\boldsymbol\\beta}_{IV} = \\mathbf Z^\\top\\mathbf y$.\n", "\n", "**Step 3.** Subtract. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · The control function reproduces 2SLS\n", "\n", "Scalar endogenous $x$, instrument vector $\\mathbf Z$, first stage $x = \\mathbf Z\\hat\\pi + \\hat v$. Prove that in the OLS regression of $y$ on $x$ and $\\hat v$, the coefficient on $x$ equals 2SLS.\n", "\n", "
Show solution\n", "\n", "**Step 1.** By FWL, the coefficient on $x$ uses $x$ residualised on $\\hat v$.\n", "\n", "**Step 2.** $\\hat v$ is orthogonal to $\\mathbf Z\\hat\\pi = \\hat x$, and $x = \\hat x + \\hat v$, so residualising $x$ on $\\hat v$ gives exactly $\\hat x$.\n", "\n", "**Step 3.** The coefficient is $\\hat x^\\top y/\\hat x^\\top\\hat x$, the 2SLS estimator. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Disagreement is what J detects\n", "\n", "With two instruments (one endogenous regressor), show that if both single-instrument estimands are equal, the population over-identifying restriction holds in the sense that some $\\beta$ satisfies both moment conditions.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Moment conditions: $E[z_j(y - \\beta x)] = 0$ for $j = 1, 2$, i.e. $\\beta = E[z_j y]/E[z_j x]$.\n", "\n", "**Step 2.** If both ratios equal the same $b$, then $\\beta = b$ solves both.\n", "\n", "**Step 3.** Conversely, if the ratios differ, no single $\\beta$ solves both, and J diverges with $n$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · No power against common invalidity\n", "\n", "Suppose each instrument has a direct effect proportional to its first stage: $\\delta_j = c\\,\\pi_j$. Show that every single-instrument estimand equals $\\beta + c$, so J has no power.\n", "\n", "
Show solution\n", "\n", "**Step 1.** From Day 62, Exercise 10: instrument $j$ gives $\\beta + \\delta_j/\\pi_j$.\n", "\n", "**Step 2.** $\\delta_j/\\pi_j = c$ for all $j$, so all estimands equal $\\beta + c$.\n", "\n", "**Step 3.** By Exercise 8, the moment conditions are jointly satisfied at $\\beta + c$: J converges to $\\chi^2_{l-k}$ as under validity. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Plausibly exogenous bounds\n", "\n", "Let $Y = \\beta X + \\gamma Z + \\varepsilon$ with $\\gamma \\in [-\\delta, \\delta]$ unknown. Show that the IV estimand is $\\beta + \\gamma/\\pi$ and that $\\beta \\in [\\beta_{IV} - \\delta/|\\pi|,\\; \\beta_{IV} + \\delta/|\\pi|]$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{Cov}(Z, Y) = \\beta\\operatorname{Cov}(Z, X) + \\gamma\\operatorname{Var}(Z)$.\n", "\n", "**Step 2.** Divide by $\\operatorname{Cov}(Z, X) = \\pi\\operatorname{Var}(Z)$: $\\beta_{IV} = \\beta + \\gamma/\\pi$.\n", "\n", "**Step 3.** $\\beta = \\beta_{IV} - \\gamma/\\pi$, which ranges over the stated interval as $\\gamma$ ranges over $[-\\delta, \\delta]$. $\\blacksquare$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Two instruments for schooling\n", "\n", "Day 64's instruments give single-instrument returns of 0.15 (SE 0.02, the quarter-of-birth indicator) and 0.10 (SE 0.04, the schooling-law indicator). Is there evidence of invalidity? What else could explain a difference?\n", "\n", "
Show solution\n", "\n", "**Step 1.** By Exercise 3, $z \\approx 1.12$: no significant disagreement.\n", "\n", "**Step 2.** Even a significant difference could reflect different LATEs: compulsory-schooling laws affect students at risk of dropping out, while quarter of birth matters for others.\n", "\n", "**Answer:** no evidence against validity; and disagreement would not prove invalidity.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · One bad instrument in a simulation\n", "\n", "In the simulation below, two instruments are valid when the second's direct effect is 0, and the second is invalid when its direct effect is 0.05 or 0.10. How do the 2SLS estimate (truth 0.5) and the J test's rejection rate change?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Direct effect 0: 2SLS ≈ 0.50 and J rejects about 5% of the time (correct size).\n", "\n", "**Step 2.** Direct effect 0.05: 2SLS ≈ 0.58; J rejects about 22%.\n", "\n", "**Step 3.** Direct effect 0.10: 2SLS ≈ 0.67; J rejects about 65%.\n", "\n", "**Answer:** J detects the bad instrument only part of the time, while the bias is already substantial: failing to reject is weak evidence of validity.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Is insurance endogenous?\n", "\n", "Using the Oregon-style lottery (Day 62), the control-function regression of ER visits on insurance and the first-stage residual gives a residual coefficient of −0.3 (SE 0.1). Interpret.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $t = -3$: insurance is endogenous.\n", "\n", "**Step 2.** The negative sign means that, holding insurance fixed, unobservables that raise insurance take-up lower ER visits (for example, people who sign up for insurance are healthier or better organised).\n", "\n", "**Answer:** OLS of ER visits on insurance would understate the effect; IV is preferred.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Plausibly exogenous college proximity\n", "\n", "Distance to college might affect wages directly (urban labour markets). Allow the direct effect on log wages to be between −0.01 and 0.01, with first stage 0.32 and IV estimate 0.131 (95% CI [0.05, 0.21]). Compute the sensitivity interval.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Shift: $\\delta/\\pi = 0.01/0.32 \\approx 0.031$.\n", "\n", "**Step 2.** Union of shifted intervals: $[0.05 - 0.031,\\; 0.21 + 0.031] = [0.019, 0.241]$.\n", "\n", "**Answer:** the return stays positive, but the range widens considerably; a weaker first stage would widen it more.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Rejection with valid instruments\n", "\n", "A study of the effect of child care subsidies on maternal employment uses two valid instruments: a lottery for subsidised places and the birth date cutoff for kindergarten eligibility. J rejects strongly. Give an explanation consistent with validity and what to report.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The lottery's compliers are mothers who applied (motivated to work); the birth-date compliers are all mothers of children near the cutoff.\n", "\n", "**Step 2.** With heterogeneous effects, the two LATEs differ, so a constant-effect J test rejects.\n", "\n", "**Answer:** report the two estimates separately, describe each set of compliers (Day 63), and avoid pooling them into one 2SLS number.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from scipy import stats\n", "\n", "rng = np.random.default_rng(66); n = 1000\n", "def tsls(y, X, Z):\n", " Xh = Z @ np.linalg.lstsq(Z, X, rcond=None)[0]\n", " return np.linalg.solve(Xh.T @ Xh, Xh.T @ y)\n", "\n", "print(\"direct effect J rejection mean 2SLS (truth 0.5)\")\n", "for d in (0, .05, .10):\n", " rej, est = 0, []\n", " for _ in range(1000):\n", " Z = rng.normal(size=(n, 2)); u = rng.normal(size=n); v = .5 * u + rng.normal(size=n)\n", " x = Z @ [.3, .3] + v; y = 1 + .5 * x + d * Z[:, 1] + u\n", " X = np.column_stack([np.ones(n), x]); ZZ = np.column_stack([np.ones(n), Z])\n", " b = tsls(y, X, ZZ); r = y - X @ b\n", " g = np.linalg.lstsq(ZZ, r, rcond=None)[0]\n", " R2 = 1 - ((r - ZZ @ g) @ (r - ZZ @ g)) / ((r - r.mean()) @ (r - r.mean()))\n", " rej += n * R2 > stats.chi2.ppf(.95, 1); est.append(b[1])\n", " print(f\"{d:13.2f} {rej / 1000:11.3f} {np.mean(est):.3f}\")\n", "\n", "# Exercise 7: the control function reproduces 2SLS\n", "Z = rng.normal(size=(n, 2)); u = rng.normal(size=n); x = Z @ [.3, .3] + .5 * u + rng.normal(size=n); y = .5 * x + u\n", "ZZ = np.column_stack([np.ones(n), Z]); vhat = x - ZZ @ np.linalg.lstsq(ZZ, x, rcond=None)[0]\n", "cf = np.linalg.lstsq(np.column_stack([np.ones(n), x, vhat]), y, rcond=None)[0]\n", "print(\"control function:\", cf[1].round(5), \" 2SLS:\", tsls(y, np.column_stack([np.ones(n), x]), ZZ)[1].round(5), \" residual coef:\", cf[2].round(3))\n", "\n", "# Exercises 2, 3 and 14\n", "print(\"J:\", 1000 * .006, \"crit\", round(stats.chi2.ppf(.95, 2), 2), \"| z:\", round(.05 / np.sqrt(.02**2 + .04**2), 2),\n", " \"| sensitivity interval:\", np.round([.05 - .01 / .32, .21 + .01 / .32], 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- With extra instruments, J = nR² tests whether the instruments agree; it cannot identify which is wrong and has no power when all are wrong in the same way.\n", "- The control-function regression reproduces 2SLS and tests whether OLS is biased.\n", "- With heterogeneous effects, disagreement can reflect different LATEs; sensitivity intervals show how much a small exclusion violation matters." ] } ] }