{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 14: Missing Data & Inverse Probability Weighting\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "MCAR, MAR and MNAR; when complete-case analysis is biased; and how weighting repairs MAR.\n", "\n", "**Learning outcomes**\n", "\n", "- Classify a missing-data mechanism as MCAR, MAR or MNAR.\n", "- Compute the bias of complete-case means and prove that IPW is unbiased under MAR.\n", "- Combine multiply imputed estimates with Rubin's rules.\n", "- Recognise causal inference as a missing-data problem for potential outcomes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The response indicator\n", "\n", "Let $R = 1$ if $Y$ is observed and $R = 0$ if it is missing; $X$ is always observed. The **mechanism** is the distribution of $R$ given the data:\n", "\n", "| Mechanism | Definition | Complete-case mean |\n", "|---|---|---|\n", "| **MCAR** (missing completely at random) | $P(R = 1 \\mid X, Y) = P(R = 1)$ | unbiased |\n", "| **MAR** (missing at random) | $P(R = 1 \\mid X, Y) = P(R = 1 \\mid X)$ | biased in general; fixable with $X$ |\n", "| **MNAR** (missing not at random) | depends on $Y$ even given $X$ | biased; needs extra assumptions |\n", "\n", "### Inverse probability weighting (IPW)\n", "\n", "Under MAR with **response propensity** $\\pi(X) = P(R = 1 \\mid X) > 0$, weighting each observed unit by $1/\\pi(X)$ rebuilds the full population:\n", "\n", "$$\n", "E\\Big[\\frac{R\\,Y}{\\pi(X)}\\Big] = E[Y].\n", "$$\n", "\n", "Two sample versions:\n", "\n", "$$\n", "\\hat\\mu_{HT} = \\frac1n\\sum_{i=1}^n \\frac{R_iY_i}{\\pi(X_i)}, \\qquad\n", "\\hat\\mu_{\\text{Hájek}} = \\frac{\\sum_i R_iY_i/\\pi(X_i)}{\\sum_i R_i/\\pi(X_i)}.\n", "$$\n", "\n", "The Hájek (normalised) version is usually more stable.\n", "\n", "### Multiple imputation and Rubin's rules\n", "\n", "Impute the missing values $m$ times from a model, estimate $\\hat Q_j$ (with variance $W_j$) in each completed data set, then combine:\n", "\n", "$$\n", "\\bar Q = \\frac1m\\sum_j\\hat Q_j, \\qquad T = \\bar W + \\Big(1 + \\frac1m\\Big)B, \\qquad B = \\frac{1}{m - 1}\\sum_j(\\hat Q_j - \\bar Q)^2.\n", "$$\n", "\n", "The between-imputation part $B$ reflects uncertainty about the missing values themselves.\n", "\n", "### Causal inference *is* a missing-data problem\n", "\n", "Each unit shows $Y(1)$ if treated and $Y(0)$ if not; the other is missing. Randomisation makes the missing potential outcome **MCAR**; unconfoundedness given $X$ makes it **MAR**; and IPW with the propensity score $e(X) = P(T = 1 \\mid X)$ is exactly today's estimator, applied twice. Phase 3 builds on this.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Spot the mechanism (1)\n", "\n", "A lab's freezer fails and destroys a random 10% of blood samples. Which mechanism?\n", "\n", "
Show solution\n", "\n", "MCAR: missingness is unrelated to anything about the patients.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Spot the mechanism (2)\n", "\n", "High earners are more likely to skip the income question, even among people with the same education and age. Which mechanism?\n", "\n", "
Show solution\n", "\n", "MNAR: missingness depends on the missing value (income) itself, even after conditioning on observed covariates.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Spot the mechanism (3)\n", "\n", "Older respondents are less likely to answer an online follow-up, but, within each age group, response is unrelated to the outcome. Which mechanism?\n", "\n", "
Show solution\n", "\n", "MAR: missingness depends only on the observed covariate (age).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · An IPW weight\n", "\n", "A respondent's group has a response rate of 25%. What weight does each respondent get, and why?\n", "\n", "
Show solution\n", "\n", "$1/0.25 = 4$: each respondent stands in for themselves and three similar non-respondents.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Complete cases under MCAR\n", "\n", "Why is the complete-case mean unbiased under MCAR? What is lost?\n", "\n", "
Show solution\n", "\n", "Respondents are a random subsample, so their mean has the same expectation as the full-sample mean. The cost is precision: a smaller $n$ means a larger standard error.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Complete-case mean under MCAR\n", "\n", "Show that $E[Y \\mid R = 1] = E[Y]$ under MCAR.\n", "\n", "
Show solution\n", "\n", "MCAR means $R$ is independent of $Y$, so $E[Y \\mid R = 1] = E[Y]$ (Day 6, Exercise 5).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · IPW is unbiased under MAR\n", "\n", "Prove $E[RY/\\pi(X)] = E[Y]$ when $P(R = 1 \\mid X, Y) = \\pi(X) > 0$.\n", "\n", "
Show solution\n", "\n", "Condition on $(X, Y)$ and use the LIE:\n", "\n", "$$\n", "E\\Big[\\frac{RY}{\\pi(X)}\\Big] = E\\Big[\\frac{Y}{\\pi(X)}E[R \\mid X, Y]\\Big] = E\\Big[\\frac{Y}{\\pi(X)}\\pi(X)\\Big] = E[Y].\n", "$$\n", "\n", "The step $E[R \\mid X, Y] = \\pi(X)$ *is* the MAR assumption, and $\\pi(X) > 0$ (overlap) is needed to divide.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · The bias under MNAR\n", "\n", "Show that $E[Y \\mid R = 1] - E[Y] = \\operatorname{Cov}(R, Y)/P(R = 1)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $E[RY] = P(R = 1)E[Y \\mid R = 1]$, so $\\operatorname{Cov}(R, Y) = P(R = 1)E[Y \\mid R = 1] - P(R = 1)E[Y]$.\n", "\n", "**Step 2.** Divide by $P(R = 1)$.\n", "\n", "**Step 3.** If response rises with $Y$, the covariance is positive and respondents overstate the mean.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Mean imputation understates variance\n", "\n", "Fill $n - r$ missing values with the mean of the $r$ observed values. Show that the sample variance of the completed data is $\\frac{r - 1}{n - 1}s_r^2$, where $s_r^2$ is the observed-data variance.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Imputed values equal the observed mean $\\bar y_r$, so the completed-data mean is still $\\bar y_r$, and they add zero to the sum of squares.\n", "\n", "**Step 2.** That sum stays $(r - 1)s_r^2$ but is now divided by $n - 1$.\n", "\n", "**Step 3.** With 30% missing, the variance shrinks by about 30% and standard errors are too small.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Rubin's rules\n", "\n", "Five imputations give estimates 10.2, 9.8, 10.5, 10.0 and 9.5, each with within-imputation variance 0.25. Find the pooled estimate and its SE.\n", "\n", "
Show solution\n", "\n", "$\\bar Q = 50/5 = 10.0$. Deviations 0.2, −0.2, 0.5, 0, −0.5 give $B = 0.58/4 = 0.145$. Then\n", "\n", "$$\n", "T = 0.25 + (1 + 1/5)(0.145) = 0.25 + 0.174 = 0.424, \\qquad \\operatorname{SE} = \\sqrt{0.424} \\approx 0.651.\n", "$$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Bias from MNAR\n", "\n", "30% of people are high earners ($Y = 100$) and 70% low earners ($Y = 40$). High earners respond with probability 0.5, low earners with 0.9. Find the true mean and the respondents' mean.\n", "\n", "
Show solution\n", "\n", "**Step 1.** True mean: $0.3(100) + 0.7(40) = 30 + 28 = 58$.\n", "\n", "**Step 2.** Share of the population who respond: high earners $0.3 \\times 0.5 = 0.15$; low earners $0.7 \\times 0.9 = 0.63$; total 0.78.\n", "\n", "**Step 3.** Respondents' mean:\n", "\n", "$$\n", "\\frac{0.15(100) + 0.63(40)}{0.78} = \\frac{15 + 25.2}{0.78} \\approx 51.5.\n", "$$\n", "\n", "**Answer:** the survey reports 51.5 instead of 58, a bias of about −6.5, because high earners are under-represented among respondents.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Repairing MAR with weights\n", "\n", "Half the sample is young (mean $Y = 20$, response rate 0.4) and half old (mean $Y = 40$, response rate 0.8); within age, response is random. Compare the complete-case mean with the IPW mean.\n", "\n", "
Show solution\n", "\n", "**Step 1.** True mean: $0.5(20) + 0.5(40) = 30$.\n", "\n", "**Step 2.** Respondents: young $0.5 \\times 0.4 = 0.2$, old $0.5 \\times 0.8 = 0.4$ (total 0.6).\n", "\n", "**Step 3.** Complete-case mean: $(0.2 \\times 20 + 0.4 \\times 40)/0.6 = 20/0.6 \\approx 33.3$ (old people over-represented).\n", "\n", "**Step 4.** IPW weights: young $1/0.4 = 2.5$, old $1/0.8 = 1.25$. Weighted mean per person in the population:\n", "\n", "$$\n", "0.2 \\times 2.5 \\times 20 + 0.4 \\times 1.25 \\times 40 = 10 + 20 = 30.\n", "$$\n", "\n", "**Answer:** complete cases give 33.3; IPW recovers the true 30.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Horvitz–Thompson versus Hájek in a household survey\n", "\n", "Four sampled households have monthly food spending $Y = (10, 20, 30, \\cdot)$ (in \\$10), where the fourth did not respond, so $R = (1, 1, 1, 0)$. Their estimated response probabilities are $\\pi = (0.5, 0.5, 0.8, 0.8)$. Compute the Horvitz–Thompson and Hájek IPW estimates of mean spending.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Weighted values $R_iY_i/\\pi_i$: $10/0.5 = 20$, $20/0.5 = 40$, $30/0.8 = 37.5$, and 0. Sum: 97.5.\n", "\n", "**Step 2.** Horvitz–Thompson divides by $n = 4$: $97.5/4 \\approx 24.4$.\n", "\n", "**Step 3.** Hájek divides by the sum of the weights, $1/0.5 + 1/0.5 + 1/0.8 = 5.25$: $97.5/5.25 \\approx 18.6$.\n", "\n", "**Answer:** 24.4 versus 18.6 (\\$244 versus \\$186). In small samples they can differ a lot; the Hájek estimate always stays within the range of the observed values (here 10–30), so it is usually preferred.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Missing outcomes in regression\n", "\n", "In a regression of wages on education, compare: (a) missingness that depends on education only; (b) missingness that depends on wages.\n", "\n", "
Show solution\n", "\n", "**(a) Missingness depends on education only**\n", "\n", "**Step 1.** Among people with the same education, respondents are a random subset, so $E[Y \\mid X, R = 1] = E[Y \\mid X]$.\n", "\n", "**Step 2.** The regression line is unchanged; you lose precision and perhaps part of the education range.\n", "\n", "**(b) Missingness depends on wages**\n", "\n", "**Step 3.** At each education level, high earners drop out. Respondents' average wage is pulled down most where wages are highest (high education).\n", "\n", "**Step 4.** That flattens the line: **attenuation bias**.\n", "\n", "**Answer:** selection on $X$ leaves the slope unbiased; selection on $Y$ biases it toward zero (see the MNAR setting in the Lab).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Potential outcomes as missing data\n", "\n", "In an RCT with 50% treated, which potential outcome is missing for each unit, what is the \"response probability\" for $Y(1)$, and which estimator of $E[Y(1)]$ does IPW give?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Treated units are missing $Y(0)$; controls are missing $Y(1)$.\n", "\n", "**Step 2.** $Y(1)$ is observed when $T = 1$, which happens with probability 0.5 for everyone: the missingness is MCAR.\n", "\n", "**Step 3.** IPW for $E[Y(1)]$: $\\frac1n\\sum_iT_iY_i/0.5$. The Hájek version divides by the realised number treated instead, giving the treated-group mean.\n", "\n", "**Step 4.** If the propensity depends on covariates, $e(X) = P(T = 1 \\mid X)$, the weights become $T/e(X)$ for $Y(1)$ and $(1 - T)/(1 - e(X))$ for $Y(0)$.\n", "\n", "**Answer:** every causal estimate fills in missing potential outcomes; with a known or estimated propensity, Step 4 is the IPW estimator of the ATE (Day 44).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(14)\n", "n = 1_000_000\n", "old = rng.random(n) < 0.5\n", "Y = np.where(old, rng.normal(40, 5, n), rng.normal(20, 5, n))\n", "pi = np.where(old, 0.8, 0.4)\n", "R = rng.random(n) < pi # MAR: depends on age only\n", "print(\"true mean :\", Y.mean().round(2))\n", "print(\"complete cases :\", Y[R].mean().round(2))\n", "print(\"IPW (HT) :\", (R * Y / pi).mean().round(2))\n", "print(\"IPW (Hajek) :\", ((R * Y / pi).sum() / (R / pi).sum()).round(2))\n", "\n", "# Rubin's rules (Exercise 10)\n", "Q = np.array([10.2, 9.8, 10.5, 10.0, 9.5]); W = 0.25\n", "B = Q.var(ddof=1); T = W + (1 + 1 / len(Q)) * B\n", "print(\"pooled\", Q.mean(), \" total var\", round(T, 3), \" SE\", round(np.sqrt(T), 3))\n", "\n", "# MNAR attenuates a regression slope\n", "educ = rng.normal(13, 2, 200_000)\n", "wage = 5 + 2 * educ + rng.normal(0, 6, 200_000)\n", "keep = rng.random(200_000) < 1 / (1 + np.exp((wage - 30) / 4)) # high earners drop out\n", "print(\"slope full:\", np.polyfit(educ, wage, 1)[0].round(3), \" slope MNAR sample:\", np.polyfit(educ[keep], wage[keep], 1)[0].round(3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- MCAR is harmless for bias; MAR is fixable with the right covariates; MNAR needs extra assumptions or bounds.\n", "- IPW weights each respondent by the inverse of its response probability; it requires MAR and overlap.\n", "- Every causal estimate fills in missing potential outcomes, so these ideas reappear throughout the course." ] } ] }