{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 12: The Delta Method\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Standard errors for functions of estimates (logs, ratios, odds, percentage effects) via a first-order Taylor expansion.\n", "\n", "**Learning outcomes**\n", "\n", "- Apply the univariate delta method to logs, square roots, inverses and odds.\n", "- Derive it from a Taylor expansion plus the CLT, and know when it fails.\n", "- Use the multivariate delta method for ratios (cost-effectiveness, relative effects, Wald IV estimates).\n", "- Convert log-point regression coefficients into percentage effects with correct standard errors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The idea\n", "\n", "You have an estimate $\\hat\\theta$ with $\\sqrt n(\\hat\\theta - \\theta) \\xrightarrow{d} N(0, \\sigma^2)$, but you care about $g(\\theta)$: a percentage effect, an elasticity, a ratio. A first-order **Taylor expansion** around $\\theta$ gives\n", "\n", "$$\n", "g(\\hat\\theta) \\approx g(\\theta) + g'(\\theta)(\\hat\\theta - \\theta),\n", "$$\n", "\n", "so $g(\\hat\\theta)$ is approximately a linear function of an approximately normal variable:\n", "\n", "$$\n", "\\sqrt n\\big(g(\\hat\\theta) - g(\\theta)\\big) \\xrightarrow{d} N\\big(0,\\; g'(\\theta)^2\\sigma^2\\big), \\qquad\n", "\\operatorname{SE}\\big(g(\\hat\\theta)\\big) \\approx |g'(\\hat\\theta)|\\,\\operatorname{SE}(\\hat\\theta).\n", "$$\n", "\n", "### Several estimates at once\n", "\n", "For a vector $\\hat\\theta$ with covariance matrix $\\Sigma$ and a scalar function $g$ with gradient $\\nabla g$:\n", "\n", "$$\n", "\\operatorname{Var}\\big(g(\\hat\\theta)\\big) \\approx \\nabla g(\\theta)^{\\top}\\,\\Sigma\\,\\nabla g(\\theta).\n", "$$\n", "\n", "For a **ratio** $R = A/B$:\n", "\n", "$$\n", "\\operatorname{Var}(\\hat R) \\approx \\frac{\\operatorname{Var}(\\hat A)}{B^2} + \\frac{A^2\\operatorname{Var}(\\hat B)}{B^4} - \\frac{2A\\operatorname{Cov}(\\hat A, \\hat B)}{B^3}.\n", "$$\n", "\n", "### Limits\n", "\n", "- It is a *local linear* approximation: poor when $g$ is very curved relative to the SE, or when $\\hat\\theta$ is near a point where $g$ blows up (a ratio with denominator near zero, e.g. a weak first stage).\n", "- If $g'(\\theta) = 0$ the first-order term vanishes and a second-order expansion is needed.\n", "- The bootstrap (Day 9) is the usual cross-check.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Derivative of the log\n", "\n", "What is $g'(x)$ for $g(x) = \\ln x$, and what does it imply for $\\operatorname{SE}(\\ln\\hat\\theta)$?\n", "\n", "
Show solution\n", "\n", "$g'(x) = 1/x$, so $\\operatorname{SE}(\\ln\\hat\\theta) \\approx \\operatorname{SE}(\\hat\\theta)/\\hat\\theta$: the coefficient of variation.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Log of a mean\n", "\n", "$\\bar X = 50$ with SE 2. Find the SE of $\\ln\\bar X$.\n", "\n", "
Show solution\n", "\n", "$2/50 = 0.04$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Square root\n", "\n", "$\\hat\\theta = 16$ with SE 0.8. Find the SE of $\\sqrt{\\hat\\theta}$.\n", "\n", "
Show solution\n", "\n", "$g'(x) = 1/(2\\sqrt x) = 1/8$, so SE $\\approx 0.8/8 = 0.1$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Inverse\n", "\n", "$\\hat\\theta = 4$ with SE 0.2. Find the SE of $1/\\hat\\theta$.\n", "\n", "
Show solution\n", "\n", "$|g'(x)| = 1/x^2 = 1/16$, so SE $\\approx 0.2/16 = 0.0125$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Odds\n", "\n", "$\\hat p = 0.6$ with SE 0.0894 (Day 11). Find the SE of the odds $\\hat p/(1 - \\hat p)$.\n", "\n", "
Show solution\n", "\n", "$g'(p) = 1/(1 - p)^2 = 1/0.16 = 6.25$, so SE $\\approx 6.25 \\times 0.0894 \\approx 0.559$ around the odds estimate 1.5.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Deriving the delta method\n", "\n", "Using a Taylor expansion with remainder and the CLT, derive the univariate delta method.\n", "\n", "
Show solution\n", "\n", "**Step 1 (mean value theorem).** For some $\\tilde\\theta$ between $\\hat\\theta$ and $\\theta$:\n", "\n", "$$\n", "g(\\hat\\theta) = g(\\theta) + g'(\\tilde\\theta)(\\hat\\theta - \\theta).\n", "$$\n", "\n", "**Step 2.** Rearrange and multiply by $\\sqrt n$:\n", "\n", "$$\n", "\\sqrt n\\big(g(\\hat\\theta) - g(\\theta)\\big) = g'(\\tilde\\theta)\\cdot\\sqrt n(\\hat\\theta - \\theta).\n", "$$\n", "\n", "**Step 3.** $\\hat\\theta \\xrightarrow{p} \\theta$ and $\\tilde\\theta$ is squeezed between them, so $\\tilde\\theta \\xrightarrow{p} \\theta$. With $g'$ continuous, $g'(\\tilde\\theta) \\xrightarrow{p} g'(\\theta)$.\n", "\n", "**Step 4.** By assumption (the CLT), $\\sqrt n(\\hat\\theta - \\theta) \\xrightarrow{d} N(0, \\sigma^2)$.\n", "\n", "**Step 5.** Slutsky's theorem (a term converging to a constant times a term converging in distribution):\n", "\n", "$$\n", "\\sqrt n\\big(g(\\hat\\theta) - g(\\theta)\\big) \\xrightarrow{d} g'(\\theta)\\,N(0, \\sigma^2) = N\\big(0,\\; g'(\\theta)^2\\sigma^2\\big). \\qquad\\blacksquare\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · The ratio formula\n", "\n", "Derive the variance of $\\hat A/\\hat B$ from the multivariate delta method.\n", "\n", "
Show solution\n", "\n", "$g(A, B) = A/B$ has gradient $\\nabla g = (1/B,\\; -A/B^2)$. Then\n", "\n", "$$\n", "\\nabla g^{\\top}\\Sigma\\nabla g = \\frac{\\operatorname{Var}(\\hat A)}{B^2} + \\frac{A^2}{B^4}\\operatorname{Var}(\\hat B) - \\frac{2A}{B^3}\\operatorname{Cov}(\\hat A, \\hat B).\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Second-order bias\n", "\n", "Using a second-order expansion, show $E[g(\\bar X)] \\approx g(\\mu) + \\tfrac12 g''(\\mu)\\sigma^2/n$, and apply it to $\\ln\\bar X$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $g(\\bar X) \\approx g(\\mu) + g'(\\mu)(\\bar X - \\mu) + \\tfrac12 g''(\\mu)(\\bar X - \\mu)^2$.\n", "\n", "**Step 2.** Take expectations: the linear term has mean zero and $E[(\\bar X - \\mu)^2] = \\sigma^2/n$.\n", "\n", "**Step 3.** For $\\ln$: $g'' = -1/\\mu^2$, so $E[\\ln\\bar X] \\approx \\ln\\mu - \\sigma^2/(2n\\mu^2)$.\n", "\n", "**Step 4.** The log of a mean is biased downward, as Jensen's inequality predicts, but the bias vanishes at rate $1/n$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · When the first derivative is zero\n", "\n", "Let $\\bar X$ be a mean with $\\mu = 0$ and variance $\\sigma^2/n$, and $g(x) = x^2$. Why does the delta method fail, and what is the limit of $n\\bar X^2/\\sigma^2$?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $g'(0) = 0$, so the first-order approximation predicts zero variance, which is wrong.\n", "\n", "**Step 2.** Instead, $\\sqrt n\\bar X/\\sigma \\xrightarrow{d} N(0, 1)$, so $n\\bar X^2/\\sigma^2 \\xrightarrow{d} \\chi^2_1$.\n", "\n", "**Step 3.** The estimator converges at rate $n$, not $\\sqrt n$, and its distribution is not normal.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Log-odds\n", "\n", "Derive the SE of the log-odds $\\ln(\\hat p/(1 - \\hat p))$ and evaluate it at $\\hat p = 0.6$, $n = 30$.\n", "\n", "
Show solution\n", "\n", "$g'(p) = 1/p + 1/(1 - p) = 1/(p(1 - p))$. With $\\operatorname{SE}(\\hat p) = \\sqrt{p(1 - p)/n}$:\n", "\n", "$$\n", "\\operatorname{SE} = \\frac{1}{\\sqrt{n\\,p(1 - p)}} = \\frac{1}{\\sqrt{30 \\times 0.24}} \\approx 0.373.\n", "$$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · From log points to percent\n", "\n", "In a log-wage regression, a training dummy has $\\hat\\beta = 0.08$ (SE 0.02). Report the exact percentage effect $100(e^{\\beta} - 1)$ with its SE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Exact percentage effect: $g(\\beta) = 100(e^{\\beta} - 1) = 100(e^{0.08} - 1) \\approx 8.33\\%$.\n", "\n", "**Step 2.** Derivative: $g'(\\beta) = 100e^{\\beta} = 108.33$.\n", "\n", "**Step 3.** Delta method: $\\operatorname{SE} \\approx 108.33 \\times 0.02 \\approx 2.17$ percentage points.\n", "\n", "**Answer:** training raises wages by 8.3% (SE 2.2 points).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Cost-effectiveness ratio\n", "\n", "A programme costs an extra $\\Delta C = 1{,}200$ (SE 300) and adds $\\Delta E = 0.4$ QALYs (SE 0.1), estimated independently. Find the ICER and its SE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Point estimate: $\\text{ICER} = 1200/0.4 = 3{,}000$ per QALY.\n", "\n", "**Step 2.** Ratio formula with zero covariance, $A = 1200$, $B = 0.4$:\n", "\n", "$$\n", "\\operatorname{Var} \\approx \\frac{\\operatorname{Var}(\\hat A)}{B^2} + \\frac{A^2\\operatorname{Var}(\\hat B)}{B^4}.\n", "$$\n", "\n", "**Step 3.** First term: $300^2/0.4^2 = 90{,}000/0.16 = 562{,}500$.\n", "\n", "**Step 4.** Second term: $1200^2 \\times 0.1^2/0.4^4 = 1{,}440{,}000 \\times 0.01/0.0256 = 562{,}500$.\n", "\n", "**Step 5.** Total $1{,}125{,}000$, so $\\operatorname{SE} \\approx 1{,}061$.\n", "\n", "**Answer:** \\$3,000 per QALY (SE ≈ \\$1,061). The imprecise denominator contributes half the variance; a bootstrap interval would be visibly skewed.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · A relative treatment effect\n", "\n", "The effect is $\\hat\\tau = 5$ (SE 1) and the control mean is $\\hat\\mu_0 = 50$ (SE 2), independent. Find the relative effect $\\hat\\tau/\\hat\\mu_0$ and its SE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Point estimate: $R = 5/50 = 0.10$.\n", "\n", "**Step 2.** Ratio formula with $A = 5$, $B = 50$, independent estimates:\n", "\n", "$$\n", "\\operatorname{Var} \\approx \\frac{1^2}{50^2} + \\frac{5^2 \\times 2^2}{50^4} = 0.0004 + 0.000016 = 0.000416.\n", "$$\n", "\n", "**Step 3.** $\\operatorname{SE} = \\sqrt{0.000416} \\approx 0.0204$.\n", "\n", "**Answer:** a 10% relative effect, ± about 2 points. Almost all the uncertainty comes from $\\hat\\tau$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · The Wald (IV) estimator\n", "\n", "An encouragement design has reduced form 0.6 (SE 0.2) and first stage 0.3 (SE 0.05), independent. Find the IV estimate and its SE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Point estimate: $\\hat\\beta = 0.6/0.3 = 2$.\n", "\n", "**Step 2.** Ratio formula with $A = 0.6$ (reduced form), $B = 0.3$ (first stage):\n", "\n", "$$\n", "\\frac{0.2^2}{0.3^2} = \\frac{0.04}{0.09} \\approx 0.444, \\qquad \\frac{0.6^2 \\times 0.05^2}{0.3^4} = \\frac{0.36 \\times 0.0025}{0.0081} \\approx 0.111.\n", "$$\n", "\n", "**Step 3.** $\\operatorname{Var} \\approx 0.556$, $\\operatorname{SE} \\approx 0.745$.\n", "\n", "**Answer:** IV estimate 2 (SE ≈ 0.75). With a weak first stage (denominator near zero) this approximation breaks down: the weak-instruments problem of Phase 5.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Log of mean income: delta method versus simulation\n", "\n", "A survey estimates mean household income (in \\$10,000) as $\\bar X$ with SE 2. Analysts report $\\ln\\bar X$. Compare the delta-method SE of $\\ln\\bar X$ with a simulation when the true mean is 50 (a rich region) and when it is 5 (a poor region). When does the delta method fail?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Delta method: $\\operatorname{SE}(\\ln\\bar X) \\approx \\operatorname{SE}/\\mu$.\n", "\n", "**Step 2.** Rich region: $2/50 = 0.04$. Poor region: $2/5 = 0.40$.\n", "\n", "**Step 3.** Simulation (code below): about 0.040 in the rich region, but about 0.52 in the poor region.\n", "\n", "**Step 4.** Reason: when $\\operatorname{SE}/\\mu$ is large, $\\bar X$ ranges over values where $\\ln$ is strongly curved, so the linear approximation is poor and $\\ln\\bar X$ is skewed.\n", "\n", "**Answer:** the delta method is excellent when the coefficient of variation is small (0.04) and misleading when it is large (0.4); check with the bootstrap or simulation.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(12)\n", "for mu, se in ((50, 2), (5, 2)):\n", " xbar = rng.normal(mu, se, 1_000_000)\n", " xbar = xbar[xbar > 0]\n", " print(f\"mu={mu}, SE={se}: delta SE of ln = {se/mu:.4f}, simulated = {np.log(xbar).std():.4f}\")\n", "\n", "# ICER (Exercise 12): delta method vs simulation\n", "dC = rng.normal(1200, 300, 1_000_000); dE = rng.normal(0.4, 0.1, 1_000_000)\n", "icer = dC / dE\n", "q25, q50, q75 = np.percentile(icer, [25, 50, 75])\n", "print(\"ICER delta SE ~ 1061; simulated robust SD (IQR/1.349):\", round((q75 - q25) / 1.349),\n", " \"; simulated 2.5/50/97.5 percentiles:\", np.percentile(icer, [2.5, 50, 97.5]).round())\n", "# (the plain SD is unstable: a few draws of dE near zero make the ratio explode)\n", "\n", "print(\"percent effect:\", round(100 * (np.exp(0.08) - 1), 2), \"SE\", round(100 * np.exp(0.08) * 0.02, 2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- $\\operatorname{SE}(g(\\hat\\theta)) \\approx |g'(\\hat\\theta)|\\operatorname{SE}(\\hat\\theta)$; for several estimates use the gradient and the covariance matrix.\n", "- Ratios with noisy denominators (ICERs, IV with a weak first stage) are where the approximation is weakest.\n", "- Check the delta method against the bootstrap when the transformation is strongly non-linear." ] } ] }