{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 70: Staggered Adoption: the Goodman-Bacon Decomposition\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "When units adopt a policy at different times, the TWFE coefficient is a weighted average of every possible 2×2 DiD, including comparisons that use already-treated units as controls. With effects that change over time those comparisons are contaminated, and TWFE can be far from, or even opposite in sign to, every true effect.\n", "\n", "**Learning outcomes**\n", "\n", "- List the 2×2 comparisons hidden inside a staggered TWFE regression.\n", "- Explain the decomposition weights and why units treated in the middle of the panel get the most weight.\n", "- Prove that TWFE is unbiased with constant effects and biased with dynamic effects through \"forbidden\" comparisons.\n", "- Compute a decomposition and compare TWFE with the true average effect." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Staggered adoption\n", "\n", "Units adopt treatment at different dates and stay treated. Write $k$ for an **early** group (adopts at $t_k$), $l$ for a **late** group ($t_l > t_k$), and $U$ for **never-treated** units. The common practice was to run the TWFE regression $Y_{it} = \\alpha_i + \\lambda_t + \\beta D_{it} + \\varepsilon_{it}$ and call $\\hat\\beta$ \"the effect\".\n", "\n", "### The decomposition\n", "\n", "Goodman-Bacon (2021) showed that, in a balanced panel,\n", "\n", "$$\n", "\\hat\\beta_{TWFE} = \\sum_{\\text{2×2 comparisons } j} s_j\\,\\hat\\beta^{2\\times2}_j, \\qquad \\sum_j s_j = 1,\\; s_j \\ge 0.\n", "$$\n", "\n", "With one early group, one late group and a never-treated group there are four comparisons:\n", "\n", "| Comparison | Treated | Control | Window |\n", "|---|---|---|---|\n", "| $\\hat\\beta_{kU}$ | early | never | all periods |\n", "| $\\hat\\beta_{lU}$ | late | never | all periods |\n", "| $\\hat\\beta^{k}_{kl}$ | early | late (not yet treated) | before $t_l$ |\n", "| $\\hat\\beta^{l}_{kl}$ | late | early (**already treated**) | from $t_k$ on |\n", "\n", "### The weights\n", "\n", "Each weight grows with the size of the two groups and with the **variance of treatment** in the comparison, $\\bar D(1 - \\bar D)$, where $\\bar D$ is the share of the window in which the treated group is treated. Groups treated in the middle of the panel ($\\bar D$ near 0.5) get the most weight; this weighting is a property of OLS, not of the policy question.\n", "\n", "### When it goes wrong\n", "\n", "- **Constant effects:** every 2×2 equals the effect; TWFE is fine.\n", "- **Effects differing across groups but constant over time:** TWFE is a convex, variance-weighted average of group effects; not the ATT, but between them.\n", "- **Effects that change over time:** the comparison $\\hat\\beta^{l}_{kl}$ uses the early group as a control after it was treated, so its still-evolving effect is subtracted. TWFE can be biased and even have the wrong sign. In the equivalent ATT language (de Chaisemartin and D'Haultfœuille), some group-time effects receive **negative weights**.\n", "\n", "### Diagnostics and fixes\n", "\n", "Plot the 2×2 estimates against their weights (a \"Bacon plot\"), check how much weight falls on already-treated controls, and use heterogeneity-robust estimators that never use treated units as controls (Day 71).\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Count the comparisons\n", "\n", "With an early group, a late group and never-treated units, list the 2×2 comparisons. How many are there with three timing groups and a never-treated group?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Two groups: each timing group against never-treated (2), and each ordered pair of timing groups with the later or earlier one as control (2). Total 4.\n", "\n", "**Step 2.** Three timing groups: 3 against never-treated, plus 2 for each of the 3 pairs = 6. Total 9 ($K^2$ with $K$ timing groups).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Treatment variance\n", "\n", "In a 10-period panel, the early group is treated from period 4, the late group from period 7. Compute $\\bar D(1 - \\bar D)$ for each against the never-treated group.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Early: treated 7 of 10 periods, $\\bar D_k = 0.7$, variance $0.21$.\n", "\n", "**Step 2.** Late: treated 4 of 10, $\\bar D_l = 0.4$, variance $0.24$.\n", "\n", "**Answer:** the late group's comparison gets slightly more weight per unit, being closer to the middle of the panel.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Constant effects\n", "\n", "If treatment raises outcomes by exactly 2 for every unit in every treated period, what does each 2×2 estimate and TWFE equal?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Each 2×2 compares changes; untreated trends are parallel and every post-period difference is 2.\n", "\n", "**Answer:** every comparison equals 2, so TWFE = 2 whatever the weights.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · The forbidden comparison\n", "\n", "Explain why the late-versus-early comparison is problematic when the early group's effect grows over time.\n", "\n", "
Show solution\n", "\n", "**Step 1.** In the window from $t_k$, the early group is always treated, so it serves as the \"control\".\n", "\n", "**Step 2.** Its outcomes keep changing because its treatment effect keeps growing, not because of a common trend.\n", "\n", "**Answer:** that growth is subtracted from the late group's change, biasing the comparison downward.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Which units identify what?\n", "\n", "In the 2×2 that compares early with late before $t_l$, why is the late group a legitimate control?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Before $t_l$, the late group is untreated: its changes reflect untreated trends.\n", "\n", "**Answer:** under parallel trends (and no anticipation by the late group), it is a valid \"not-yet-treated\" control.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Constant effects mean no bias\n", "\n", "Assume parallel trends and a constant effect $\\tau$ for all units and periods. Prove each 2×2 estimand equals $\\tau$, and hence $\\beta_{TWFE} = \\tau$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** In any 2×2, the treated group's change is (common trend) + $\\tau$ × (change in its treatment status between pre and post windows) = trend + $\\tau$.\n", "\n", "**Step 2.** The control group's change is the common trend: for not-yet-treated controls nothing changes; for already-treated controls the effect $\\tau$ is present in both windows and cancels.\n", "\n", "**Step 3.** Each 2×2 equals $\\tau$; the weights sum to 1. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · The contaminated comparison\n", "\n", "Let the early group's effect be $\\tau_k(e)$ at event time $e$ and the late group's $\\tau_l(e)$. Show that, under parallel trends, $\\beta^{l}_{kl} = \\overline{\\tau_l}^{\\,post} - \\big(\\overline{\\tau_k}^{\\,post} - \\overline{\\tau_k}^{\\,pre}\\big)$, where the bars average over the windows' periods.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Late group: its change between the window's pre part ($t_k \\le t < t_l$, untreated) and post part ($t \\ge t_l$) is trend + average $\\tau_l$ over the post part.\n", "\n", "**Step 2.** Early group: treated throughout; its change is trend + (average $\\tau_k$ in the post part − average $\\tau_k$ in the pre part).\n", "\n", "**Step 3.** Subtract. $\\blacksquare$ If $\\tau_k$ grows over time, the bracket is positive and the comparison is biased downwards.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Group-specific constant effects\n", "\n", "Suppose effects differ across groups but not over time ($\\tau_k$, $\\tau_l$). Show $\\beta_{TWFE} = (s_{kU} + s^{k}_{kl})\\tau_k + (s_{lU} + s^{l}_{kl})\\tau_l$, a convex combination that generally differs from the ATT.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\beta_{kU} = \\tau_k$, $\\beta_{lU} = \\tau_l$, $\\beta^{k}_{kl} = \\tau_k$ (late group untreated in that window).\n", "\n", "**Step 2.** By Exercise 7 with constant $\\tau_k$, the bracket is 0, so $\\beta^{l}_{kl} = \\tau_l$.\n", "\n", "**Step 3.** Collect terms; weights are non-negative and sum to 1. $\\blacksquare$ The ATT weights groups by their number of treated unit-periods instead.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · A wrong sign\n", "\n", "Construct an example (early effect growing, late effect small) in which $\\beta^{l}_{kl} < 0$ although every unit's effect is non-negative.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Let $\\tau_k(e) = e$ (zero on impact, growing by 1 per period) and $\\tau_l(e) = 0.5$.\n", "\n", "**Step 2.** With the early group adopting at period 4 and the late at 7 in a 10-period panel, the window pre part is periods 4–6 ($e = 0, 1, 2$; mean 1) and post part 7–10 ($e = 3, \\dots, 6$; mean 4.5).\n", "\n", "**Step 3.** By Exercise 7: $\\beta^{l}_{kl} = 0.5 - (4.5 - 1) = -3$.\n", "\n", "**Answer:** a negative comparison from non-negative effects; with enough weight it can make TWFE negative. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Weights favour the middle\n", "\n", "For a timing group compared with never-treated units of equal size, the weight is proportional to $\\bar D(1 - \\bar D)$. Show this is maximised when treatment covers half the panel.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $f(\\bar D) = \\bar D - \\bar D^2$ has derivative $1 - 2\\bar D$.\n", "\n", "**Step 2.** Zero at $\\bar D = 0.5$; second derivative $-2 < 0$. $\\blacksquare$ Groups adopting near the start or end of the panel get little weight, regardless of how policy-relevant they are.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Divorce law reforms with dynamic effects\n", "\n", "States adopt a reform in period 4 (early) or 7 (late) of a 10-period panel; a third of states never adopt; groups are equal in size. Effects grow: 1 on impact, +0.5 per year. The 2×2 estimates are 2.5 (early vs never), 1.75 (late vs never), 1.5 (early vs not-yet-treated late) and 0 (late vs already-treated early), with weights 0.318, 0.364, 0.136, 0.182. Compute TWFE and compare with the ATT over all treated state-years.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\hat\\beta_{TWFE} = 0.318(2.5) + 0.364(1.75) + 0.136(1.5) + 0.182(0) = 0.795 + 0.637 + 0.204 + 0 \\approx 1.64$.\n", "\n", "**Step 2.** ATT: early cells (periods 4–10, $e = 0, \\dots, 6$) average $1 + 0.5 \\times 3 = 2.5$; late cells (7–10, $e = 0, \\dots, 3$) average $1.75$.\n", "\n", "**Step 3.** Weighted by treated cells: $(7 \\times 2.5 + 4 \\times 1.75)/11 = 24.5/11 \\approx 2.23$.\n", "\n", "**Answer:** TWFE (1.64) understates the ATT (2.23) by about a quarter; the forbidden comparison contributes 0 with 18% weight.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Changing the group sizes\n", "\n", "Repeat Exercise 11 with group sizes 2 (early) : 1 (late) : 3 (never), which give weights 0.525, 0.300, 0.075, 0.100. Compute TWFE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $0.525(2.5) + 0.300(1.75) + 0.075(1.5) + 0.100(0) = 1.3125 + 0.525 + 0.1125 = 1.95$.\n", "\n", "**Answer:** 1.95, against an ATT of 2.33 with these group sizes: more never-treated units shift weight to clean comparisons, but bias remains.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Diagnosing a published estimate\n", "\n", "A paper's TWFE estimate of a hospital-merger effect on prices is 1%; a Bacon decomposition shows 40% of the weight on comparisons with already-merged hospitals as controls, whose 2×2 estimates are −4%. What would you conclude?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The clean comparisons must average about $(1 - 0.4 \\times (-4))/0.6 = 2.6/0.6 \\approx 4.3\\%$.\n", "\n", "**Step 2.** The negative forbidden comparisons pull TWFE down, consistent with price effects that grow after mergers.\n", "\n", "**Answer:** the true effect is likely several times larger than 1%; re-estimate with a heterogeneity-robust method (Day 71).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Electronic health records\n", "\n", "Hospitals adopted electronic health records between 2008 and 2016, with incentives starting in 2011; almost all had adopted by 2016. A study wants the effect on medication errors. What design concerns does staggered adoption raise?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Few never-treated hospitals: most comparisons in TWFE would use earlier adopters as controls for later ones.\n", "\n", "**Step 2.** Learning effects make the impact grow over time, the case where those comparisons are biased.\n", "\n", "**Step 3.** Better: group-time ATTs with not-yet-treated controls, restricted to periods before almost everyone adopted, and event-study plots by cohort.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · What to report\n", "\n", "List what a staggered-adoption DiD paper should report alongside (or instead of) a single TWFE coefficient.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The adoption timeline (how many units adopt when; any never-treated units).\n", "\n", "**Step 2.** A Bacon decomposition: estimates and weights by comparison type.\n", "\n", "**Step 3.** Heterogeneity-robust estimates (group-time ATTs and event-study aggregations) and their pre-trends.\n", "\n", "**Step 4.** A discussion of which average effect is reported and for which units.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "def panel(T, tk, tl, eff, eff_late=None):\n", " eff_late = eff_late or eff\n", " t = np.arange(1, T + 1)\n", " D = np.array([(t >= tk), (t >= tl), np.zeros(T, bool)]).astype(float)\n", " base = np.array([1.0, 2.0, 0.5])[:, None] + 0.3 * t[None, :]\n", " E = np.array([eff(t - tk), eff_late(t - tl), np.zeros(T)])\n", " return base + D * E, D, t\n", "\n", "def twfe_weighted(Y, D, w):\n", " w = w / w.sum()\n", " dm = lambda M: M - M.mean(1, keepdims=True) - (w[:, None] * M).sum(0) + (w[:, None] * M.mean(1, keepdims=True)).sum()\n", " Dd = dm(D)\n", " return (w[:, None] * Dd * Y).sum() / (w[:, None] * Dd ** 2).sum()\n", "\n", "def bacon(Y, t, tk, tl, w):\n", " did = lambda a, b, pre, post: (Y[a, post].mean() - Y[a, pre].mean()) - (Y[b, post].mean() - Y[b, pre].mean())\n", " pk, pl = t >= tk, t >= tl\n", " b = np.array([did(0, 2, ~pk, pk), did(1, 2, ~pl, pl), did(0, 1, ~pk & ~pl, pk & ~pl), did(1, 0, pk & ~pl, pl)])\n", " nk, nl, nu = w / w.sum(); Dk, Dl = pk.mean(), pl.mean()\n", " sh = lambda a, c: a / (a + c)\n", " s = np.array([(nk + nu) ** 2 * sh(nk, nu) * (1 - sh(nk, nu)) * Dk * (1 - Dk),\n", " (nl + nu) ** 2 * sh(nl, nu) * (1 - sh(nl, nu)) * Dl * (1 - Dl),\n", " ((nk + nl) * (1 - Dl)) ** 2 * sh(nk, nl) * (1 - sh(nk, nl)) * (Dk - Dl) / (1 - Dl) * (1 - Dk) / (1 - Dl),\n", " ((nk + nl) * Dk) ** 2 * sh(nk, nl) * (1 - sh(nk, nl)) * Dl / Dk * (Dk - Dl) / Dk])\n", " return b, s / s.sum()\n", "\n", "for label, eff in [(\"constant effect 2\", lambda e: 2 + 0 * e), (\"dynamic 1 + 0.5e\", lambda e: 1 + .5 * e)]:\n", " for w in (np.array([1., 1, 1]), np.array([2., 1, 3])):\n", " Y, D, t = panel(10, 4, 7, eff)\n", " b, s = bacon(Y, t, 4, 7, w)\n", " att = (w[0] * eff(np.arange(7)).sum() + w[1] * eff(np.arange(4)).sum()) / (7 * w[0] + 4 * w[1])\n", " print(f\"{label:18s} sizes {w.astype(int)}: 2x2s {b.round(2)} weights {s.round(3)} \"\n", " f\"sum {(s * b).sum():.3f} TWFE {twfe_weighted(Y, D, w):.3f} ATT {att:.3f}\")\n", "\n", "# Exercise 9: early effect = e (growing), late effect = 0.5\n", "Y, D, t = panel(10, 4, 7, lambda e: e * 1.0, lambda e: 0.5 + 0 * e)\n", "b, s = bacon(Y, t, 4, 7, np.ones(3))\n", "print(\"late vs already-treated early:\", round(b[3], 2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Staggered TWFE averages all 2×2 DiDs, with OLS weights that favour groups treated mid-panel.\n", "- Comparisons that use already-treated units as controls subtract their evolving effects; with dynamic effects TWFE is biased and can flip sign.\n", "- Report the adoption pattern and a decomposition, and use estimators that only compare with not-yet- or never-treated units." ] } ] }