{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 34: The Fundamental Problem & Selection Bias\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Why naive comparisons fail. Decomposing them into the ATE, selection bias and heterogeneity bias, and the Roy model of people choosing treatment.\n", "\n", "**Learning outcomes**\n", "\n", "- Decompose a naive comparison into ATE + selection bias + heterogeneity bias.\n", "- Use the Roy model to explain selection on levels versus selection on gains.\n", "- Predict the sign of selection bias in common economic and health settings.\n", "- Recognise Ashenfelter's dip and why before–after comparisons mislead." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Two decompositions\n", "\n", "With $Y = Y(0) + T\\tau$ (Day 31) and $p = P(T = 1)$:\n", "\n", "$$\n", "\\underbrace{E[Y \\mid T = 1] - E[Y \\mid T = 0]}_{\\text{naive}} = \\text{ATT} + \\underbrace{E[Y(0) \\mid T = 1] - E[Y(0) \\mid T = 0]}_{\\text{selection bias}}\n", "$$\n", "\n", "$$\n", "\\phantom{E[Y \\mid T = 1] - E[Y \\mid T = 0]} = \\text{ATE} + \\text{selection bias} + \\underbrace{(1 - p)(\\text{ATT} - \\text{ATC})}_{\\text{heterogeneity bias}}.\n", "$$\n", "\n", "- **Selection bias** (selection on levels): the treated would have had different outcomes even untreated.\n", "- **Heterogeneity bias** (selection on gains): the treated benefit more or less than the untreated.\n", "\n", "Randomisation sets both to zero (Day 35).\n", "\n", "### The Roy model\n", "\n", "Roy (1951) asked how workers choose occupations. Applied to programmes: unit $i$ chooses treatment if its gain exceeds its cost,\n", "\n", "$$\n", "T_i = \\mathbf 1\\{Y_i(1) - Y_i(0) > c_i\\}.\n", "$$\n", "\n", "Choice now depends on potential outcomes, the opposite of an experiment. Two typical patterns:\n", "\n", "- **Positive selection on gains:** those who gain most enrol, so ATT > ATE > ATC.\n", "- **Negative selection on levels:** programmes target, or attract, people with poor prospects: $E[Y(0) \\mid T = 1] < E[Y(0) \\mid T = 0]$, so naive comparisons understate effects (training for the unemployed, hospital care for the sick).\n", "\n", "If gains are normal, $\\tau \\sim N(\\mu, \\sigma^2)$, and $T = \\mathbf 1\\{\\tau > 0\\}$, then\n", "\n", "$$\n", "\\text{ATT} = \\mu + \\sigma\\frac{\\phi(\\mu/\\sigma)}{\\Phi(\\mu/\\sigma)}.\n", "$$\n", "\n", "### Ashenfelter's dip\n", "\n", "Trainees' earnings often *fall* just before they enrol (they enrol because they lost a job). A before–after comparison then credits the programme with the natural recovery (Ashenfelter, 1978).\n", "\n", "## Part 1 · Warm-up\n", "\n", "A programme has ATT = 2, ATC = 1, 40% of the population treated, and selection bias −3 (all in \\$1,000 of earnings)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · ATE\n", "\n", "Compute the ATE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** ATE $= p\\cdot\\text{ATT} + (1 - p)\\cdot\\text{ATC}$ (Day 33).\n", "\n", "**Step 2.** $= 0.4(2) + 0.6(1) = 1.4$.\n", "\n", "**Answer:** \\$1,400.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · The naive comparison\n", "\n", "What does the naive comparison report?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Naive $=$ ATT $+$ selection bias $= 2 + (-3)$.\n", "\n", "**Answer:** −1: the programme looks harmful.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Three-term check\n", "\n", "Verify the three-term decomposition.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Heterogeneity bias $= (1 - p)(\\text{ATT} - \\text{ATC}) = 0.6(2 - 1) = 0.6$.\n", "\n", "**Step 2.** ATE $+$ SB $+$ HB $= 1.4 - 3 + 0.6 = -1$. ✓\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Sign of selection bias\n", "\n", "For each, predict the sign of $E[Y(0) \\mid T = 1] - E[Y(0) \\mid T = 0]$: (a) earnings, for a programme for displaced workers; (b) health, for people who choose to take vitamins; (c) test scores, for pupils whose parents pay for tutoring.\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a) Displaced workers would earn less anyway: **negative**.\n", "\n", "**Step 2.** (b) Vitamin takers tend to be healthier and wealthier: **positive**.\n", "\n", "**Step 3.** (c) Families who pay for tutoring are richer and more engaged: **positive** (unless tutoring is bought for struggling pupils: then negative).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · The Roy rule\n", "\n", "Workers enrol if $Y(1) - Y(0) > 1$. Worker A has $Y(0) = 10$, $Y(1) = 13$; worker B has $Y(0) = 20$, $Y(1) = 20.5$. Who enrols?\n", "\n", "
Show solution\n", "\n", "**Step 1.** A's gain $3 > 1$: enrols.\n", "\n", "**Step 2.** B's gain $0.5 < 1$: does not.\n", "\n", "**Answer:** A enrols. Note that A also has lower $Y(0)$: here selection on gains and negative selection on levels go together.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · The two-term decomposition\n", "\n", "Prove naive $=$ ATT $+$ selection bias.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Observed outcomes: $E[Y \\mid T = 1] = E[Y(1) \\mid T = 1]$ and $E[Y \\mid T = 0] = E[Y(0) \\mid T = 0]$.\n", "\n", "**Step 2.** Add and subtract $E[Y(0) \\mid T = 1]$:\n", "\n", "$$\n", "\\big(E[Y(1) \\mid T = 1] - E[Y(0) \\mid T = 1]\\big) + \\big(E[Y(0) \\mid T = 1] - E[Y(0) \\mid T = 0]\\big).\n", "$$\n", "\n", "**Step 3.** The first bracket is the ATT; the second is the selection bias. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · The three-term decomposition\n", "\n", "Prove ATT $=$ ATE $+ (1 - p)(\\text{ATT} - \\text{ATC})$, and hence the three-term decomposition.\n", "\n", "
Show solution\n", "\n", "**Step 1.** From ATE $= p\\,\\text{ATT} + (1 - p)\\,\\text{ATC}$: $\\text{ATT} - \\text{ATE} = \\text{ATT} - p\\,\\text{ATT} - (1 - p)\\,\\text{ATC} = (1 - p)(\\text{ATT} - \\text{ATC})$.\n", "\n", "**Step 2.** Substitute ATT $=$ ATE $+ (1 - p)(\\text{ATT} - \\text{ATC})$ into Exercise 6. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Randomisation kills both biases\n", "\n", "Prove that if $T$ is independent of $(Y(0), Y(1))$, selection bias and heterogeneity bias are both zero.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Independence of $T$ and $Y(0)$: $E[Y(0) \\mid T = 1] = E[Y(0) \\mid T = 0]$, so selection bias $= 0$.\n", "\n", "**Step 2.** Independence of $T$ and $\\tau = Y(1) - Y(0)$: ATT $=$ ATC $=$ ATE, so heterogeneity bias $= 0$.\n", "\n", "**Step 3.** Hence naive $=$ ATE. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Selection on gains raises the ATT\n", "\n", "In the Roy model with $T = \\mathbf 1\\{\\tau > c\\}$ (and $0 < P(T = 1) < 1$), prove ATT $\\ge$ ATE $\\ge$ ATC.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Every treated unit has $\\tau > c$, so ATT $= E[\\tau \\mid \\tau > c] > c$.\n", "\n", "**Step 2.** Every untreated unit has $\\tau \\le c$, so ATC $= E[\\tau \\mid \\tau \\le c] \\le c$.\n", "\n", "**Step 3.** So ATT $>$ ATC, and ATE $= p\\,\\text{ATT} + (1 - p)\\,\\text{ATC}$ lies between them. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · The Roy ATT with normal gains\n", "\n", "If $\\tau \\sim N(\\mu, \\sigma^2)$ and $T = \\mathbf 1\\{\\tau > 0\\}$, prove ATT $= \\mu + \\sigma\\phi(\\mu/\\sigma)/\\Phi(\\mu/\\sigma)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Standardise: $\\tau = \\mu + \\sigma Z$, and $\\tau > 0 \\iff Z > -\\mu/\\sigma$.\n", "\n", "**Step 2.** Truncated-normal mean: $E[Z \\mid Z > a] = \\phi(a)/(1 - \\Phi(a))$ with $a = -\\mu/\\sigma$.\n", "\n", "**Step 3.** By symmetry, $\\phi(-\\mu/\\sigma) = \\phi(\\mu/\\sigma)$ and $1 - \\Phi(-\\mu/\\sigma) = \\Phi(\\mu/\\sigma)$.\n", "\n", "**Step 4.** ATT $= \\mu + \\sigma E[Z \\mid Z > a] = \\mu + \\sigma\\phi(\\mu/\\sigma)/\\Phi(\\mu/\\sigma)$. $\\blacksquare$\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Roy model for a training programme\n", "\n", "Gains from training (in \\$1,000) are normally distributed with mean 2 and sd 3 across workers, and workers enrol only if they gain. Compute the share who enrol, the ATT and the ATE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Share enrolling: $P(\\tau > 0) = \\Phi(2/3) = \\Phi(0.667) \\approx 0.748$.\n", "\n", "**Step 2.** $\\phi(0.667) = 0.3989\\,e^{-0.667^2/2} \\approx 0.3989 \\times 0.801 \\approx 0.319$.\n", "\n", "**Step 3.** ATT $= 2 + 3 \\times 0.319/0.748 \\approx 2 + 1.28 = 3.28$.\n", "\n", "**Step 4.** ATE $= \\mu = 2$.\n", "\n", "**Answer:** 75% enrol; ATT ≈ \\$3,280 versus ATE \\$2,000. An evaluation of current trainees overstates what compulsory training would achieve.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Ashenfelter's dip\n", "\n", "Trainees usually earn \\$15k a year. In the year before training, after losing jobs, they earn \\$12k. After training they earn \\$16k. Without training, their earnings would have recovered to \\$15k. Compare the before–after estimate with the true effect.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Before–after: $16 - 12 = 4$.\n", "\n", "**Step 2.** True effect: $Y(1) - Y(0) = 16 - 15 = 1$.\n", "\n", "**Step 3.** The extra 3 is the recovery from the dip (mean reversion) that would have happened anyway.\n", "\n", "**Answer:** before–after says \\$4k; the truth is \\$1k. Comparing with a pre-dip baseline or with a matched control group that also lost jobs avoids this.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Do hospitals harm health?\n", "\n", "In a health survey, people admitted to hospital last year report worse health (3.2 on a 1–5 scale, 5 = excellent) than those not admitted (3.9). Decompose the naive difference if hospital care truly raises health by 0.3 for those admitted.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Naive $= 3.2 - 3.9 = -0.7$.\n", "\n", "**Step 2.** Naive $=$ ATT $+$ selection bias, with ATT $= +0.3$.\n", "\n", "**Step 3.** Selection bias $= -0.7 - 0.3 = -1.0$.\n", "\n", "**Answer:** admitted patients would have reported health a full point lower without care; the sick go to hospital. The naive comparison makes beneficial care look harmful.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · The college wage premium\n", "\n", "College graduates earn 60% more than high-school graduates (in logs, 0.47). A twins study suggests ability and family background account for 0.15 log points. Decompose the raw gap.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Raw gap $= 0.47$ = ATT $+$ selection bias, with selection bias $= 0.15$ (graduates would have earned more anyway).\n", "\n", "**Step 2.** ATT $= 0.47 - 0.15 = 0.32$, i.e. $e^{0.32} - 1 \\approx 38\\%$.\n", "\n", "**Answer:** about two-thirds of the raw premium is a causal return for graduates; one-third is selection. If gains are heterogeneous, the effect for non-graduates (ATC) may be smaller still.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Which direction?\n", "\n", "A city opens a free clinic, and clinic users have lower blood pressure than non-users. An analyst concludes the clinic works. List the threats using the decomposition, and propose a design that avoids them.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Selection on levels: health-conscious people use clinics and would have lower blood pressure anyway (positive bias in the naive \"effect\", i.e. too good).\n", "\n", "**Step 2.** Selection on levels the other way: people with symptoms seek care (negative bias). The net sign is unknown.\n", "\n", "**Step 3.** Selection on gains: those who expect to benefit attend, so even an unbiased ATT would overstate the ATE of universal access.\n", "\n", "**Step 4.** Design: randomise invitations or outreach (an encouragement design, Day 41), or exploit a clinic opening date that varies across neighbourhoods (difference-in-differences, Phase 5).\n", "\n", "**Answer:** the naive comparison mixes the clinic's effect with who chooses to attend; randomised encouragement isolates it.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from scipy import stats\n", "\n", "# Roy model with normal gains (Exercise 11)\n", "mu, sd = 2.0, 3.0\n", "p = stats.norm.cdf(mu / sd)\n", "att = mu + sd * stats.norm.pdf(mu / sd) / p\n", "print(f\"share enrolling {p:.3f}, ATT {att:.3f}, ATE {mu}\")\n", "\n", "rng = np.random.default_rng(34)\n", "n = 1_000_000\n", "y0 = rng.normal(20, 5, n)\n", "tau = rng.normal(mu, sd, n)\n", "T = tau > 0 # select on gains only\n", "y = y0 + T * tau\n", "naive = y[T].mean() - y[~T].mean()\n", "sb = y0[T].mean() - y0[~T].mean()\n", "att_s, atc_s = tau[T].mean(), tau[~T].mean()\n", "print(f\"simulated ATT {att_s:.3f}, ATC {atc_s:.3f}, ATE {tau.mean():.3f}\")\n", "print(f\"naive {naive:.3f} = ATE {tau.mean():.3f} + SB {sb:.3f} + HB {(1 - T.mean()) * (att_s - atc_s):.3f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Naive comparison = ATE + selection bias + heterogeneity bias.\n", "- In the Roy model people select on gains, so ATT > ATE > ATC; programmes for the disadvantaged also select on levels.\n", "- Before–after comparisons are vulnerable to Ashenfelter's dip and mean reversion." ] } ] }