{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 33: Estimands: ATE, ATT, ATC & CATE\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Which average effect answers which policy question, how the averages relate, and why the distribution of effects is mostly out of reach.\n", "\n", "**Learning outcomes**\n", "\n", "- Define and compute ATE, ATT, ATC and CATE, and link them with the law of iterated expectations.\n", "- Prove ATE = p·ATT + (1 − p)·ATC and state when ATT = ATE.\n", "- Match estimands to policy questions (keep, expand, target a programme).\n", "- Distinguish quantile treatment effects from quantiles of individual effects, and relative from absolute effects." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The main averages\n", "\n", "With $\\tau_i = Y_i(1) - Y_i(0)$ and covariates $X$:\n", "\n", "| Estimand | Definition | Policy question |\n", "|---|---|---|\n", "| ATE | $E[\\tau]$ | What if *everyone* were treated versus no one? |\n", "| ATT | $E[\\tau \\mid T = 1]$ | Is the programme worth it for those who use it now? |\n", "| ATC | $E[\\tau \\mid T = 0]$ | What would expanding it to current non-users achieve? |\n", "| CATE | $\\tau(x) = E[\\tau \\mid X = x]$ | For whom does it work? Whom should we target? |\n", "\n", "They are linked by the LIE (Day 6):\n", "\n", "$$\n", "\\text{ATE} = E[\\tau(X)], \\qquad \\text{ATT} = E[\\tau(X) \\mid T = 1], \\qquad \\text{ATE} = p\\cdot\\text{ATT} + (1 - p)\\cdot\\text{ATC},\n", "$$\n", "\n", "where $p = P(T = 1)$ and the second equality uses $E[\\tau \\mid X, T = 1] = \\tau(X)$, which holds when treatment is unconfounded given $X$ (Day 42). The averages coincide when effects are constant, or more generally when who is treated is unrelated to how much they gain.\n", "\n", "### Effects on distributions\n", "\n", "The **quantile treatment effect** $\\text{QTE}(q) = F_{Y(1)}^{-1}(q) - F_{Y(0)}^{-1}(q)$ compares the quantiles of the two outcome *distributions*. It is identified by a randomised experiment. It is **not** the $q$-th quantile of individual effects $\\tau_i$, which depends on the joint distribution of $(Y(1), Y(0))$, never observed (Day 31, Exercise 15).\n", "\n", "### Absolute versus relative effects\n", "\n", "For a binary outcome with risks $R_1 = P(Y(1) = 1)$ and $R_0 = P(Y(0) = 1)$:\n", "\n", "- **Risk difference** $R_1 - R_0$ (the ATE); its inverse is the **number needed to treat**.\n", "- **Risk ratio** $R_1/R_0$; **vaccine efficacy** $= 1 - R_1/R_0$.\n", "\n", "The same intervention can look large on one scale and small on another.\n", "\n", "## Part 1 · Warm-up\n", "\n", "A training programme's effect on annual earnings (\\$1,000) depends on prior earnings $X$: $\\tau(\\text{low}) = 3$ and $\\tau(\\text{high}) = 1$. 40% of workers are low earners." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · ATE from CATEs\n", "\n", "Compute the ATE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** ATE $= E[\\tau(X)] = P(\\text{low})\\tau(\\text{low}) + P(\\text{high})\\tau(\\text{high})$.\n", "\n", "**Step 2.** $= 0.4(3) + 0.6(1) = 1.2 + 0.6 = 1.8$.\n", "\n", "**Answer:** \\$1,800.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · ATT\n", "\n", "80% of low earners and 20% of high earners enrol. Assuming enrolment is unconfounded given $X$, compute the ATT.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Share enrolled: $P(T = 1) = 0.4(0.8) + 0.6(0.2) = 0.32 + 0.12 = 0.44$.\n", "\n", "**Step 2.** Composition of enrollees: $P(\\text{low} \\mid T = 1) = 0.32/0.44 \\approx 0.727$; high $\\approx 0.273$.\n", "\n", "**Step 3.** ATT $= 0.727(3) + 0.273(1) \\approx 2.18 + 0.27 = 2.45$.\n", "\n", "**Answer:** ≈ \\$2,450: enrollees are disproportionately the workers who gain most.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · ATC\n", "\n", "Compute the ATC.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Non-enrollees: $1 - 0.44 = 0.56$; low earners among them $0.4 \\times 0.2 = 0.08$, so $P(\\text{low} \\mid T = 0) = 0.08/0.56 \\approx 0.143$.\n", "\n", "**Step 2.** ATC $= 0.143(3) + 0.857(1) \\approx 0.43 + 0.86 = 1.29$.\n", "\n", "**Answer:** ≈ \\$1,290.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Check the identity\n", "\n", "Verify ATE $= p\\cdot$ATT $+ (1 - p)\\cdot$ATC.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $0.44(2.455) + 0.56(1.286) \\approx 1.080 + 0.720$.\n", "\n", "**Step 2.** $= 1.80$. ✓\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Vaccine efficacy\n", "\n", "In a trial, 1% of vaccinated and 5% of placebo participants are infected. Compute the risk difference, vaccine efficacy and the number needed to vaccinate to prevent one infection.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Risk difference $= 0.01 - 0.05 = -0.04$.\n", "\n", "**Step 2.** Efficacy $= 1 - 0.01/0.05 = 0.80$.\n", "\n", "**Step 3.** Number needed to vaccinate $= 1/0.04 = 25$.\n", "\n", "**Answer:** 4 fewer infections per 100 vaccinated; 80% efficacy; 25 vaccinations per infection prevented.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · ATE is a weighted average of ATT and ATC\n", "\n", "Prove ATE $= p\\cdot$ATT $+ (1 - p)\\cdot$ATC.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Law of total expectation over $T$: $E[\\tau] = E[\\tau \\mid T = 1]P(T = 1) + E[\\tau \\mid T = 0]P(T = 0)$.\n", "\n", "**Step 2.** Substitute the definitions: $\\text{ATE} = p\\cdot\\text{ATT} + (1 - p)\\cdot\\text{ATC}$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · ATT through CATEs\n", "\n", "Assume $E[\\tau \\mid X, T = 1] = \\tau(X)$ (unconfoundedness). Prove ATT $= E[\\tau(X) \\mid T = 1]$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** LIE within the treated: $\\text{ATT} = E[\\tau \\mid T = 1] = E\\big[E[\\tau \\mid X, T = 1] \\mid T = 1\\big]$.\n", "\n", "**Step 2.** Replace the inner expectation by $\\tau(X)$: $\\text{ATT} = E[\\tau(X) \\mid T = 1]$. $\\blacksquare$\n", "\n", "ATT averages the CATEs over the treated's covariate distribution; ATE averages them over everyone's.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · When ATT equals ATE\n", "\n", "Prove ATT $=$ ATE if (a) effects are constant, or (b) $T$ is independent of $\\tau$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a) If $\\tau_i = c$ for all $i$, every average of $\\tau$ equals $c$.\n", "\n", "**Step 2.** (b) Independence means conditioning on $T$ does not change the distribution of $\\tau$: $E[\\tau \\mid T = 1] = E[\\tau]$.\n", "\n", "**Answer:** randomised experiments satisfy (b), so their treated group's average effect is the ATE. Self-selection on gains breaks (b).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Median of differences versus difference of medians\n", "\n", "Three workers have $(Y(0), Y(1))$ = (1, 2), (2, 6), (3, 3). Compute the difference in medians and the median of individual effects.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $Y(0)$: 1, 2, 3, median 2. $Y(1)$: 2, 6, 3, median 3. Difference of medians: $3 - 2 = 1$.\n", "\n", "**Step 2.** Individual effects: $2 - 1 = 1$, $6 - 2 = 4$, $3 - 3 = 0$. Median of effects: 1.\n", "\n", "**Step 3.** Now keep both marginal distributions but pair them differently: (1, 3), (2, 2), (3, 6). Effects 2, 0, 3: median 2, while the difference of medians is still 1.\n", "\n", "**Step 4.** Another pairing, (1, 6), (2, 3), (3, 2), gives effects 5, 1, −1: one worker in three is now *harmed*.\n", "\n", "**Answer:** the difference in medians (the QTE at 0.5) is 1 whatever the pairing; the median effect and the share harmed depend on the unobservable pairing.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Heterogeneity beyond covariates\n", "\n", "Prove $\\operatorname{Var}(\\tau_i) \\ge \\operatorname{Var}(\\tau(X))$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Law of total variance (Day 6): $\\operatorname{Var}(\\tau) = E[\\operatorname{Var}(\\tau \\mid X)] + \\operatorname{Var}(E[\\tau \\mid X])$.\n", "\n", "**Step 2.** The first term is non-negative, and $E[\\tau \\mid X] = \\tau(X)$.\n", "\n", "**Step 3.** So $\\operatorname{Var}(\\tau) \\ge \\operatorname{Var}(\\tau(X))$. $\\blacksquare$\n", "\n", "CATEs reveal only the heterogeneity *explained by* $X$; people with the same $X$ can still respond differently.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Keep, expand or target?\n", "\n", "Using Exercises 1–3, a ministry asks three questions: (a) Is the current programme worth its cost of \\$2,000 per trainee? (b) Should it be made compulsory for all unemployed workers? (c) Should it be offered only to low earners? Which estimand answers each, and what is the answer?\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a) Current trainees: ATT ≈ \\$2,450 > \\$2,000: worth keeping.\n", "\n", "**Step 2.** (b) Everyone: ATE = \\$1,800 < \\$2,000: compulsory roll-out fails the cost test.\n", "\n", "**Step 3.** (c) Low earners: CATE = \\$3,000 > \\$2,000, while for high earners \\$1,000 < \\$2,000: target low earners.\n", "\n", "**Answer:** ATT, ATE and CATE give three different answers, each correct for its question.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Extending to non-users\n", "\n", "A microfinance programme has ATT = 15% higher business profits among current borrowers. A donor wants to fund outreach to non-borrowers. Why might the ATC be much smaller, and how could you estimate it?\n", "\n", "
Show solution\n", "\n", "**Step 1.** People borrow when they have profitable projects: selection on gains (Day 34) makes ATT > ATC.\n", "\n", "**Step 2.** The ATC requires $E[Y(1) \\mid T = 0]$, never observed.\n", "\n", "**Step 3.** Ways forward: an encouragement RCT among non-borrowers (identifies effects for those induced to borrow, Day 41), or, under unconfoundedness, CATEs averaged over non-borrowers' covariates.\n", "\n", "**Answer:** expect ATC < ATT. Randomised studies of microcredit generally find modest average effects.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · A minimum wage and the wage distribution\n", "\n", "After a minimum-wage rise, the 10th percentile of wages rises by \\$1.50 and the median by \\$0.10. Can we conclude that the lowest-paid 10% of workers gained \\$1.50 each?\n", "\n", "
Show solution\n", "\n", "**Step 1.** These are quantile treatment effects: differences between quantiles of two distributions.\n", "\n", "**Step 2.** Workers can change rank: some low earners may lose jobs (dropping out of the wage distribution), others may be pushed above the 10th percentile.\n", "\n", "**Step 3.** So the person at the 10th percentile afterwards need not be the person who was there before.\n", "\n", "**Answer:** no. QTEs describe how the distribution shifted, not individual gains, unless one assumes ranks are preserved (rank invariance).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Effects on log wages\n", "\n", "A programme raises wages by 10% for half the workers and 0% for the rest. Wages are \\$10/hour without the programme for everyone. Compute the ATE on wages, the ATE on log wages, and $\\ln E[Y(1)] - \\ln E[Y(0)]$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $Y(1)$ is 11 for half and 10 for half: ATE on levels $= 0.5(1) = 0.50$ dollars per hour.\n", "\n", "**Step 2.** ATE on logs $= 0.5\\ln(1.1) \\approx 0.5(0.0953) = 0.0477$.\n", "\n", "**Step 3.** Log of means: $\\ln(10.5) - \\ln(10) \\approx 0.0488$.\n", "\n", "**Answer:** \\$0.50; 0.0477; 0.0488. The average of log effects is not the log of the average effect (Jensen); with heterogeneous effects, state the scale of the estimand.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Same trial, different headlines\n", "\n", "A statin trial reports 10-year heart-attack risk 2% (treated) versus 3% (placebo). Write three headlines: relative risk reduction, absolute risk reduction, and number needed to treat.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Relative: $1 - 0.02/0.03 \\approx 33\\%$ fewer heart attacks.\n", "\n", "**Step 2.** Absolute: $0.03 - 0.02 = 1$ percentage point.\n", "\n", "**Step 3.** NNT: $1/0.01 = 100$ people treated for 10 years to prevent one heart attack.\n", "\n", "**Answer:** \"a third fewer heart attacks\", \"1 point lower risk\" and \"100 treated per attack prevented\" describe the same result. Cost-effectiveness needs the absolute effect.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(33)\n", "n = 1_000_000\n", "low = rng.random(n) < 0.4\n", "tau = np.where(low, 3.0, 1.0)\n", "T = rng.random(n) < np.where(low, 0.8, 0.2) # enrolment depends on X only\n", "print(f\"ATE {tau.mean():.3f} ATT {tau[T].mean():.3f} ATC {tau[~T].mean():.3f} p {T.mean():.3f}\")\n", "print(\"check p*ATT + (1-p)*ATC:\", round(T.mean() * tau[T].mean() + (1 - T.mean()) * tau[~T].mean(), 3))\n", "\n", "# Vaccine and statin metrics\n", "for name, r1, r0 in ((\"vaccine\", 0.01, 0.05), (\"statin\", 0.02, 0.03)):\n", " print(f\"{name}: RD {r1 - r0:+.3f}, relative reduction {1 - r1 / r0:.0%}, NNT {1 / (r0 - r1):.0f}\")\n", "\n", "# Logs versus levels (Exercise 14)\n", "y0 = np.full(2, 10.0); y1 = np.array([11.0, 10.0])\n", "print(\"ATE levels\", (y1 - y0).mean(), \" ATE logs\", round(np.mean(np.log(y1) - np.log(y0)), 4),\n", " \" log of means\", round(np.log(y1.mean()) - np.log(y0.mean()), 4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- ATE, ATT, ATC and CATE answer different policy questions; choose the estimand before the method.\n", "- ATE = p·ATT + (1 − p)·ATC; they differ when people select on their gains.\n", "- Distributional and relative effects need care: quantile effects are not individual effects, and scale matters." ] } ] }