Day 80: Phase 5 Capstone: Choosing a Design · Math for Causal Inference

Theory

Start from the variation

The first question is not "which estimator?" but "what moves treatment for reasons unrelated to outcomes?"

Source of variation	Design	Estimand	Key assumption	Diagnostics
You randomise	RCT (Phase 3)	ATE (or ITT)	randomisation, no attrition bias	balance, attrition, compliance
A random or as-good-as-random nudge to take-up	IV (61–66)	LATE for compliers	independence, exclusion, monotonicity	first stage F, balance on Z, AR intervals, complier profile
A policy changes for some groups at some time	DiD / event study (67–72)	ATT for treated groups	parallel trends, no anticipation	pre-trends, placebo outcomes, sensitivity bounds, robust staggered estimators
A rule assigns treatment at a threshold	RD (73–77)	effect at the cutoff (compliers there if fuzzy)	continuity at the cutoff	density test, covariate balance, placebo cutoffs, bandwidth sensitivity
A rule changes the slope of a schedule	RKD (78)	marginal effect at the kink	smoothness	slope placebo, density derivative
None of the above is credible	bounds (79), sensitivity (95)	identified set	only what you state	transparency about assumptions

Estimands differ

Each design answers a question about a particular group: compliers with a lottery, treated states, students at a test-score cutoff. With heterogeneous effects, credible designs applied to the same policy can legitimately disagree. Before comparing numbers, compare estimands.

Combining designs

Wald-DiD (fuzzy DiD): if a policy raises take-up in treated groups, divide the DiD in outcomes by the DiD in take-up.
Difference in discontinuities: if a jump already existed at a cutoff before a policy started applying there, subtract the old jump from the new one.
RD + bounds, IV + DiD, event studies with instruments: assumptions combine, and so do diagnostics.

A workflow

State the causal question and the estimand (whose effect, of what, on what, when).
Describe the assignment mechanism with institutional detail; draw the DAG.
Choose the design that matches the variation; state its assumptions plainly.
Run the diagnostics; report the reduced form or raw comparisons graphically.
Report estimates with robust inference and sensitivity analyses.
Discuss external validity: for whom does the estimate speak?

Part 1 · Warm-up

Exercise 1 · Match the design

Name the design: (a) A city lets residents enter a lottery for housing vouchers. (b) A province raises its minimum wage; neighbouring provinces do not. (c) Children born after 1 September start school a year later. (d) A disability benefit replaces 60% of earnings up to a cap. (e) An NGO enrols villages based on undocumented criteria, with no comparison group.

Show solution

Step 1. (a) IV (lottery as instrument; ITT plus LATE).

Step 2. (b) DiD / event study.

Step 3. (c) RD on birth date (fuzzy, as some parents defer).

Step 4. (d) RKD at the cap.

Step 5. (e) Bounds or selection on observables with sensitivity analysis; no clean design.

Exercise 2 · What can be checked?

For IV, DiD and RD, name one implication of the key assumption that can be checked in data and one part that cannot.

Show solution

Step 1. IV: balance of pre-determined covariates across instrument values (checkable); exclusion (not checkable).

Step 2. DiD: parallel pre-trends (checkable); parallel post-treatment trends (not).

Step 3. RD: density and covariate continuity (checkable); continuity of potential outcomes themselves (not directly).

Exercise 3 · Three numbers for one programme

A school-meals programme is evaluated three ways: an RD at the eligibility income cutoff (+0.12 sd in test scores), a DiD across districts that adopted early (+0.08), and a lottery among oversubscribed schools (+0.15 per pupil fed). Should they agree?

Show solution

Step 1. RD: effect for pupils at the income cutoff.

Step 2. DiD: ATT for pupils in early-adopting districts (a mix of incomes).

Step 3. Lottery IV: LATE for pupils in oversubscribed schools who ate because their school won.

Answer: not necessarily; with heterogeneous effects they estimate different averages. Disagreement is informative about heterogeneity, provided each design passes its diagnostics.

Exercise 4 · Name the estimand

Give the estimand of each: a sharp RD; a fuzzy RD; a Wald estimator with one-sided non-compliance; a 2×2 DiD; Lee bounds.

Show solution

Step 1. Sharp RD: ATE at the cutoff.

Step 2. Fuzzy RD: LATE for compliers at the cutoff.

Step 3. One-sided Wald: ATT (effect on the treated).

Step 4. 2×2 DiD: ATT for the treated group in the post period.

Step 5. Lee bounds: an interval for the effect on always-observed units.

Exercise 5 · A diagnostics list

For a staggered DiD of a state paid-leave law, list four diagnostics.

Show solution

Step 1. Event-study pre-trends using a heterogeneity-robust estimator.

Step 2. Bacon decomposition of TWFE to see forbidden comparisons.

Step 3. Placebo outcomes (groups not covered by the law).

Step 4. Sensitivity to bounded violations of parallel trends (breakdown value).

Part 2 · Proofs

Exercise 6 · Wald-DiD

Suppose a reform raises take-up $D$ more in group 1 than in group 0, and $Y = \alpha_g + \lambda_t + \tau D + \varepsilon$ with a constant effect $\tau$ and parallel trends in $\varepsilon$ . Prove that the DiD in $Y$ divided by the DiD in $D$ equals $\tau$ .

Show solution

Step 1. Group-period means: $\bar Y_{gt} = \alpha_g + \lambda_t + \tau\bar D_{gt} + \bar\varepsilon_{gt}$ .

Step 2. DiD in $Y$ : the $\alpha_g$ and $\lambda_t$ cancel; parallel trends cancels the $\varepsilon$ terms; left with $\tau\times$ DiD in $D$ .

Step 3. Divide (DiD in $D$ non-zero). $\blacksquare$ With heterogeneous effects, extra assumptions are needed for a LATE interpretation (de Chaisemartin and D'Haultfœuille).

Exercise 7 · Difference in discontinuities

A cutoff at age 65 had a jump $J_0$ in employment before a pension reform (from other age-65 rules) and $J_1$ after the reform also started at 65. Show that $J_1 - J_0$ identifies the reform's effect at 65 if the other rules' jump is unchanged.

Show solution

Step 1. $J_0 = \delta$ (other rules); $J_1 = \delta + \tau$ (other rules plus reform).

Step 2. $J_1 - J_0 = \tau$ when $\delta$ is the same in both periods. $\blacksquare$ This combines RD's local comparison with DiD's removal of a stable confounder.

Exercise 8 · Placebo outcomes

Show that under parallel trends, a DiD on an outcome that treatment cannot affect has expectation zero, and explain what a non-zero placebo implies.

Show solution

Step 1. For an unaffected outcome, $Y(1) = Y(0)$ , so the ATT is zero.

Step 2. By Day 67, Exercise 9, the DiD equals ATT + trend bias = trend bias.

Answer: if parallel trends holds for the placebo, its DiD is zero in expectation; a non-zero placebo reveals differential trends, which may also affect the real outcome (if the outcomes share trend drivers). $\blacksquare$

Exercise 9 · OLS or IV by mean squared error

OLS has bias $b$ and variance $v_O$ ; IV is unbiased with variance $v_I > v_O$ . Show IV has lower MSE iff $b^2 > v_I - v_O$ .

Show solution

Step 1. $\text{MSE}_{OLS} = b^2 + v_O$ ; $\text{MSE}_{IV} = v_I$ .

Step 2. $v_I < b^2 + v_O \iff b^2 > v_I - v_O$ . $\blacksquare$ With a weak instrument, $v_I$ can be so large that a biased OLS is closer to the truth, but its bias is unknown, so IV's confidence interval is the honest summary.

Exercise 10 · Reweighting for external validity

Suppose effects vary only with a covariate $X$ : $\tau(x)$ , and a study identifies $\tau(x)$ for each $x$ in its sample. Show the ATE for a target population with covariate distribution $Q$ is $\int\tau(x)\,dQ(x)$ , and that the study's own average uses its own distribution $P$ .

Show solution

Step 1. Target ATE: $E_Q[Y(1) - Y(0)] = E_Q[E[Y(1) - Y(0) \mid X]] = E_Q[\tau(X)]$ .

Step 2. The study's average is $E_P[\tau(X)]$ ; they differ when $P \ne Q$ and $\tau$ varies.

Answer: reweighting the study's conditional effects to the target's covariate mix transports the estimate, if effects depend only on observed $X$ (and $Q$ 's support is covered). $\blacksquare$

Part 3 · Applied

Exercise 11 · Minimum wages: two designs

One study compares contiguous border counties across state lines (DiD with county pairs); another uses the bunching of wages at the new minimum to count jobs lost below it. What does each assume, and why might they disagree?

Show solution

Step 1. Border-pair DiD: parallel trends between neighbouring counties; estimates employment effects for those counties.

Step 2. Bunching: jobs that disappear below the new minimum versus excess jobs at and above it, relative to a counterfactual wage distribution; assumes the counterfactual distribution can be predicted (e.g., from other states).

Answer: they use different variation and estimands; disagreement may come from spillovers up the wage distribution or from local labour-market differences. Each paper's diagnostics, not its design label, decide credibility.

Exercise 12 · Class size: RD versus an experiment

The Tennessee STAR experiment found that classes about 7 pupils smaller raised scores by about 0.2 sd; Maimonides' rule RD finds about 0.25 points per pupil on a different test in Israel. How should these be compared?

Show solution

Step 1. Put both in comparable units (sd per pupil): STAR ≈ 0.03 sd per pupil; convert the RD estimate using the test's sd.

Step 2. Estimands differ: STAR's ATE for kindergarten to grade 3 in Tennessee; the RD's effect at the 40-pupil threshold in Israeli schools.

Answer: agreement in standardised units across designs increases confidence; differences may reflect age, baseline class size and context rather than bias.

Exercise 13 · Compulsory schooling: IV or RD?

Evaluate two designs for the effect of an extra year of schooling: (a) IV using changes in compulsory-schooling laws across states and cohorts; (b) RD on school-entry birth-date cutoffs.

Show solution

Step 1. (a) LATE for students who stay in school only because of the law (likely low-attainment); threats: other policies changing with the laws, weak first stages.

Step 2. (b) Compares children born just before and after the cutoff: they differ in school-starting age and in age at test, not only in years of schooling; exclusion is problematic for "years of schooling".

Answer: (a) speaks to compulsory schooling policy for dropouts; (b) is better suited to the effect of school-starting age.

Exercise 14 · A staggered hospital payment reform

A country moves hospitals from fee-for-service to bundled payments, region by region over four years, plus one region that never switches. Design the evaluation.

Show solution

Step 1. Estimand: ATT on cost per episode and readmissions, by years since adoption.

Step 2. Design: group-time ATTs (Callaway–Sant'Anna) with never-treated and not-yet-treated regions; event-study aggregation over a balanced window.

Step 3. Diagnostics: pre-trends, placebo outcomes (conditions not covered by bundles), Bacon decomposition of any TWFE estimate, sensitivity to bounded violations.

Step 4. Inference clustered by region (few clusters: wild bootstrap or randomisation inference over adoption dates).

Exercise 15 · One world, three estimands

In the simulation below, a programme's effect is larger for poorer households: $\tau(x) = 2 - 1.5x$ for a poverty score $x \in [-1, 1]$ (lower is poorer). Households with $x < 0$ are eligible; a lottery among eligible households offers places, and poorer households comply more. Compute the population ATE, the ATE among eligible households, the RD estimate at the eligibility cutoff (using a hypothetical universal take-up rule), and the lottery's LATE.

Show solution

Step 1. Population ATE: $2 - 1.5E[x] = 2$ .

Step 2. Eligible ATE: $2 - 1.5(-0.5) = 2.75$ .

Step 3. RD at the cutoff: $\tau(0) = 2$ .

Step 4. Lottery LATE: compliers are poorer than the average eligible household, so the LATE exceeds 2.75 (the code finds about 2.9).

Answer: four correct numbers for four questions. Policy makers need the one matching the decision: expanding eligibility (RD-type), scaling up within the eligible (eligible ATE), or improving outreach (LATE-type).

Verify in Python

import numpy as np

rng = np.random.default_rng(80); n = 2_000_000
x = rng.uniform(-1, 1, n)
tau = 2 - 1.5 * x
y0 = 5 + x + rng.normal(0, 1, n)
eligible = x < 0
print(f"population ATE {tau.mean():.3f}   eligible ATE {tau[eligible].mean():.3f}")

# Sharp RD at the cutoff with universal take-up for the eligible
d_rd = eligible.astype(float); y = y0 + d_rd * tau
def side(xx, yy, h=.2):
    w = np.clip(1 - np.abs(xx) / h, 0, None); m = w > 0
    X = np.column_stack([np.ones(m.sum()), xx[m]]); W = w[m]
    return np.linalg.solve(X.T @ (W[:, None] * X), X.T @ (W * yy[m]))[0]
rd = side(-x[eligible], y[eligible]) - side(x[~eligible], y[~eligible])
print(f"RD at cutoff {rd:.3f} (tau(0) = 2)")

# Lottery among the eligible; poorer households comply more
e = eligible
Z = rng.random(n) < .5
comply = rng.random(n) < (0.3 - 0.6 * x)          # poorer households (x near -1) comply with prob 0.9, those near 0 with 0.3
D = e & Z & comply
yL = y0 + D * tau
wald = (yL[e & Z].mean() - yL[e & ~Z].mean()) / (D[e & Z].mean() - D[e & ~Z].mean())
print(f"lottery LATE {wald:.3f}  (true complier average {tau[e & comply].mean():.3f})")

# Exercise 9: OLS vs IV by MSE
b, vO, vI = .05, .0001, .0016
print("IV preferred by MSE:", b ** 2 > vI - vO)

Key takeaways

Start from the source of exogenous variation; it determines the design, its assumptions and its estimand.
Every design has testable implications; run them, plot the raw comparisons and report robust inference.
Credible designs can disagree because they target different people; designs can also be combined (Wald-DiD, difference in discontinuities) when one alone is not enough.

Day 80. Phase 5 Capstone: Choosing a Design

Theory

Start from the variation

Estimands differ

Combining designs

A workflow

Part 1 · Warm-up

Exercise 1 · Match the design

Exercise 2 · What can be checked?

Exercise 3 · Three numbers for one programme

Exercise 4 · Name the estimand

Exercise 5 · A diagnostics list

Part 2 · Proofs

Exercise 6 · Wald-DiD

Exercise 7 · Difference in discontinuities

Exercise 8 · Placebo outcomes

Exercise 9 · OLS or IV by mean squared error

Exercise 10 · Reweighting for external validity

Part 3 · Applied

Exercise 11 · Minimum wages: two designs

Exercise 12 · Class size: RD versus an experiment

Exercise 13 · Compulsory schooling: IV or RD?

Exercise 14 · A staggered hospital payment reform

Exercise 15 · One world, three estimands

Verify in Python

Key takeaways

Simulation Lab: Choosing a Design