{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 42: Unconfoundedness & Overlap\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "The two assumptions behind every \"selection on observables\" method, what they buy, how to probe them, and what to do when overlap is poor.\n", "\n", "**Learning outcomes**\n", "\n", "- State unconfoundedness and overlap and prove that together they identify the ATE (and, more weakly, the ATT).\n", "- Diagnose covariate imbalance with normalised differences and overlap with propensity-score histograms.\n", "- Use placebo outcomes to probe unconfoundedness and understand what trimming does to the estimand.\n", "- Judge the plausibility of selection on observables in economic and health applications." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### From experiments to observational data\n", "\n", "Without randomisation, treatment depends on choices. **Selection on observables** assumes that, once we compare units with the same covariates $X$, assignment is as good as random.\n", "\n", "**Unconfoundedness** (conditional independence, ignorability):\n", "\n", "$$\n", "\\big(Y(0), Y(1)\\big) \\perp T \\mid X.\n", "$$\n", "\n", "**Overlap** (positivity, common support): with the propensity score $e(x) = P(T = 1 \\mid X = x)$,\n", "\n", "$$\n", "0 < e(x) < 1 \\quad\\text{for all } x.\n", "$$\n", "\n", "Together they identify the ATE through the adjustment formula (Day 6):\n", "\n", "$$\n", "E[Y(1)] = E\\big[E[Y \\mid T = 1, X]\\big], \\qquad E[Y(0)] = E\\big[E[Y \\mid T = 0, X]\\big].\n", "$$\n", "\n", "For the **ATT**, it is enough that $Y(0) \\perp T \\mid X$ and $e(x) < 1$: we only need to impute the treated's untreated outcomes.\n", "\n", "### Unconfoundedness is untestable…\n", "\n", "It concerns $Y(0)$ and $Y(1)$, which are never jointly observed. Its plausibility rests on knowing *how* treatment was assigned: which variables officials, doctors or applicants used. It is most credible when $X$ includes everything the decision-maker saw, especially **lagged outcomes** (past earnings, past test scores).\n", "\n", "### …but can be probed\n", "\n", "- **Placebo outcomes:** estimate the \"effect\" on an outcome measured *before* treatment (e.g. earnings two years earlier, not used as a control). A non-zero estimate signals hidden confounding.\n", "- **Multiple control groups:** results should not depend on which comparison group is used.\n", "- **Sensitivity analysis** (Day 95): how strong would an unobserved confounder need to be to explain the result away?\n", "\n", "### Overlap diagnostics\n", "\n", "- **Normalised differences:** $\\Delta = (\\bar x_1 - \\bar x_0)/\\sqrt{(s_1^2 + s_0^2)/2}$. Values above about 0.25 signal that simple regression adjustment will rely on extrapolation (Imbens & Rubin 2015).\n", "- **Propensity-score histograms** by treatment group: regions with almost only treated or only control units are where comparisons are impossible.\n", "- **Trimming:** dropping units with $\\hat e(x) \\notin [0.1, 0.9]$ (Crump et al. 2009) improves precision but changes the estimand to the effect for the overlap population.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · A normalised difference\n", "\n", "Trainees earned \\$10k on average before the programme (sd 6k); a comparison group from a household survey earned \\$15k (sd 8k). Compute the normalised difference.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Pooled sd: $\\sqrt{(36 + 64)/2} = \\sqrt{50} \\approx 7.07$.\n", "\n", "**Step 2.** $\\Delta = (10 - 15)/7.07 \\approx -0.71$.\n", "\n", "**Answer:** −0.71, far above the 0.25 rule of thumb: the groups are very different.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Extreme weights\n", "\n", "A control unit has $e(x) = 0.98$. If it is used to impute the treated's untreated outcomes with weight $e/(1 - e)$ (Day 44), what weight does it get?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $0.98/0.02 = 49$.\n", "\n", "**Answer:** this one control stands in for 49 treated units; estimates become fragile. Poor overlap shows up as extreme weights.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · The adjustment formula\n", "\n", "Smokers and non-smokers are offered a cessation app. Among young adults (60% of the population), app users quit at 30% and non-users at 20%; among older adults, 25% and 10%. Assuming unconfoundedness given age, compute the ATE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Effect by age: young $0.30 - 0.20 = 0.10$; older $0.25 - 0.10 = 0.15$.\n", "\n", "**Step 2.** ATE $= 0.6(0.10) + 0.4(0.15) = 0.06 + 0.06 = 0.12$.\n", "\n", "**Answer:** 12 percentage points.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Trimming\n", "\n", "10% of units have $\\hat e < 0.1$ and 5% have $\\hat e > 0.9$. After trimming, what share of the sample remains, and for whom is the effect estimated?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Remaining: $1 - 0.10 - 0.05 = 85\\%$.\n", "\n", "**Answer:** 85%; the estimate is the average effect for units whose covariates make both treatment and control plausible, not the ATE for everyone.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · What to condition on\n", "\n", "To study the effect of a microfinance loan on household consumption, which variables belong in $X$: (a) baseline income, (b) household size at baseline, (c) business profits after the loan, (d) whether a relative is a loan officer?\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a), (b): pre-treatment and likely related to both borrowing and consumption: include.\n", "\n", "**Step 2.** (d): pre-treatment and affects access to loans (and maybe consumption via networks): include.\n", "\n", "**Step 3.** (c): post-treatment and affected by the loan: exclude (bad control).\n", "\n", "**Answer:** (a), (b), (d).\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Identification of the ATE\n", "\n", "Prove that unconfoundedness and overlap imply $E[Y(1)] = E[E[Y \\mid T = 1, X]]$, and similarly for $Y(0)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** LIE: $E[Y(1)] = E\\big[E[Y(1) \\mid X]\\big]$.\n", "\n", "**Step 2.** Unconfoundedness: $E[Y(1) \\mid X] = E[Y(1) \\mid T = 1, X]$.\n", "\n", "**Step 3.** Consistency: among the treated $Y = Y(1)$, so $E[Y(1) \\mid T = 1, X] = E[Y \\mid T = 1, X]$.\n", "\n", "**Step 4.** Overlap ($e(X) > 0$) guarantees treated units exist at every $X$, so this conditional mean is defined.\n", "\n", "**Step 5.** Combine: $E[Y(1)] = E[E[Y \\mid T = 1, X]]$. The $Y(0)$ case uses $e(X) < 1$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Weaker conditions for the ATT\n", "\n", "Prove that $Y(0) \\perp T \\mid X$ and $e(X) < 1$ suffice to identify the ATT.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\text{ATT} = E[Y \\mid T = 1] - E[Y(0) \\mid T = 1]$; the first term is observed.\n", "\n", "**Step 2.** LIE over the treated's covariates: $E[Y(0) \\mid T = 1] = E\\big[E[Y(0) \\mid X, T = 1] \\mid T = 1\\big]$.\n", "\n", "**Step 3.** $Y(0) \\perp T \\mid X$ gives $E[Y(0) \\mid X, T = 1] = E[Y(0) \\mid X, T = 0] = E[Y \\mid X, T = 0]$.\n", "\n", "**Step 4.** $e(X) < 1$ guarantees control units exist at every $X$ found among the treated.\n", "\n", "**Answer:** $\\text{ATT} = E[Y \\mid T = 1] - E\\big[E[Y \\mid X, T = 0] \\mid T = 1\\big]$. Nothing is assumed about $Y(1)$ or about $e(X) > 0$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Overlap failure means extrapolation\n", "\n", "Suppose $e(x^*) = 1$ for some covariate value $x^*$ with positive probability. Show that $E[Y(0) \\mid X = x^*]$ is not identified without further assumptions.\n", "\n", "
Show solution\n", "\n", "**Step 1.** At $x^*$ every unit is treated, so no $Y(0)$ is ever observed there.\n", "\n", "**Step 2.** Two data-generating processes that agree everywhere except on $E[Y(0) \\mid X = x^*]$ produce identical observable data.\n", "\n", "**Step 3.** So the data cannot distinguish them: not identified. A regression model fills the gap only by extrapolating its functional form.\n", "\n", "**Answer:** without overlap, estimates rest on modelling assumptions, not on comparisons.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Placebo outcomes\n", "\n", "Let $Y^{pre}$ be an outcome measured before treatment, so the true effect on it is zero. Show that if unconfoundedness held for $Y^{pre}$ (given $X$ excluding $Y^{pre}$), the adjusted \"effect\" on $Y^{pre}$ would be zero, and explain the logic of the test.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Treatment cannot affect the past: $Y^{pre}(1) = Y^{pre}(0)$.\n", "\n", "**Step 2.** If $Y^{pre} \\perp T \\mid X$, the adjustment formula gives $E[E[Y^{pre} \\mid T = 1, X]] - E[E[Y^{pre} \\mid T = 0, X]] = 0$.\n", "\n", "**Step 3.** A non-zero estimate shows that, given $X$, treated and controls still differ systematically.\n", "\n", "**Answer:** the placebo checks for confounding of the kind that also affects earlier outcomes; passing it is reassuring, not proof.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Bad controls break unconfoundedness\n", "\n", "Training $T$ is randomised, and it raises the chance of taking a full-time job $F$. Show that comparing wages of trained and untrained workers *within* $F = 1$ can be biased even though $T$ was random.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $F$ has potential values $F(1), F(0)$. Trained full-timers have $F(1) = 1$; untrained full-timers have $F(0) = 1$.\n", "\n", "**Step 2.** Write $E[Y \\mid T = 1, F = 1] - E[Y \\mid T = 0, F = 1]$. With randomised $T$ this equals $E[Y(1) \\mid F(1) = 1] - E[Y(0) \\mid F(0) = 1]$.\n", "\n", "**Step 3.** Workers who get full-time jobs only with training are likely less skilled than those who would get them anyway, so the two groups differ in $Y(0)$.\n", "\n", "**Answer:** conditioning on a post-treatment variable compares different populations; it breaks the independence that randomisation created.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Survey comparison groups for a training programme\n", "\n", "An evaluation compares 185 training participants with 2,490 household-survey respondents. Normalised differences: age −0.9, education −0.7, baseline earnings −1.6, married −1.2. What do these imply, and what should the analyst do before estimating?\n", "\n", "
Show solution\n", "\n", "**Step 1.** All differences far exceed 0.25: the survey respondents are much older, better educated, richer and more often married.\n", "\n", "**Step 2.** Many survey respondents have covariate values with almost no participants (propensity near 0): weak overlap.\n", "\n", "**Step 3.** A linear regression would extrapolate across these gaps.\n", "\n", "**Answer:** estimate propensity scores, inspect overlap, trim or match to comparable respondents, and check results with placebo outcomes. This is the lesson of LaLonde (1986) and Dehejia & Wahba (1999).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Hospital quality\n", "\n", "A ministry ranks hospitals by 30-day mortality, adjusting for age, sex and diagnosis. Why might unconfoundedness fail, and what data would help?\n", "\n", "
Show solution\n", "\n", "**Step 1.** \"Treatment\" is the hospital attended; sicker patients may go to specialist hospitals because of severity not captured by diagnosis codes.\n", "\n", "**Step 2.** Unobserved severity raises mortality and affects which hospital is chosen: unconfoundedness given the coded $X$ fails, penalising good specialist hospitals.\n", "\n", "**Step 3.** Richer clinical data (lab values, vital signs on arrival), or quasi-random assignment such as ambulance rotation or distance to hospitals, would help.\n", "\n", "**Answer:** adjusted rankings are only as good as the severity measures.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · What trimming estimates\n", "\n", "In a study of a cash-transfer programme's effect on child schooling, households with $\\hat e > 0.9$ are the poorest (almost all treated). After trimming them, what estimand do you get, and how should it be reported?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The poorest households are removed, so the estimate is the average effect among moderately poor households where both treated and untreated exist.\n", "\n", "**Step 2.** If effects are largest for the poorest, the trimmed estimate understates the ATE.\n", "\n", "**Answer:** report it as the effect for the overlap population, describe who was trimmed, and avoid presenting it as the programme-wide ATE.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · How strong would a confounder need to be?\n", "\n", "An observational study finds that preschool attendance raises later earnings by 8%. Critics point to unmeasured parental motivation. Describe the logic of a sensitivity analysis.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Posit a confounder $U$ and parametrise its strength: how much it raises the odds of attending, and how much it raises earnings.\n", "\n", "**Step 2.** For each pair of strengths, compute the bias (Day 34) and the corrected estimate.\n", "\n", "**Step 3.** Report the smallest confounder that would make the effect zero, and compare it with the strength of observed covariates (e.g. \"as strong as parental education\").\n", "\n", "**Answer:** sensitivity analysis replaces \"untestable\" with \"how wrong would we have to be\" (Day 95).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · When treatment follows a rule\n", "\n", "A subsidy is given to every household whose poverty score is below 30. Is unconfoundedness given the score plausible? Is overlap?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Treatment is a deterministic function of the score, so given the score there is no other selection: unconfoundedness holds by design.\n", "\n", "**Step 2.** But $e(\\text{score}) = 1$ below 30 and 0 above: overlap fails everywhere.\n", "\n", "**Step 3.** Comparisons are possible only near the cut-off, by assuming outcomes would be continuous in the score there.\n", "\n", "**Answer:** unconfoundedness yes, overlap no: this is the regression discontinuity design (Days 73–77).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(42)\n", "n = 20_000\n", "x = rng.normal(0, 1, n) # e.g. standardised baseline earnings\n", "for strength in (0.5, 1.5, 3.0):\n", " e = 1 / (1 + np.exp(-strength * x))\n", " T = rng.random(n) < e\n", " nd = (x[T].mean() - x[~T].mean()) / np.sqrt((x[T].var() + x[~T].var()) / 2)\n", " out = np.mean((e < 0.1) | (e > 0.9))\n", " w = e[~T] / (1 - e[~T])\n", " print(f\"selection strength {strength}: normalised diff {nd:.2f}, share outside [0.1, 0.9] {out:.2f}, \"\n", " f\"max ATT weight on a control {w.max():.0f}\")\n", "\n", "# Adjustment formula (Exercise 3)\n", "print(\"ATE by age adjustment:\", round(0.6 * (0.30 - 0.20) + 0.4 * (0.25 - 0.10), 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Unconfoundedness plus overlap identify the ATE; the ATT needs only the conditions on Y(0) and e(x) < 1.\n", "- Unconfoundedness is untestable but can be probed with placebo outcomes and multiple comparison groups.\n", "- Check overlap first; trimming improves precision but changes the estimand." ] } ] }