Day 34: The Fundamental Problem & Selection Bias

Theory

Two decompositions

With $Y = Y(0) + T\tau$ (Day 31) and $p = P(T = 1)$ :

\underbrace{E[Y \mid T = 1] - E[Y \mid T = 0]}_{\text{naive}} = \text{ATT} + \underbrace{E[Y(0) \mid T = 1] - E[Y(0) \mid T = 0]}_{\text{selection bias}}

\phantom{E[Y \mid T = 1] - E[Y \mid T = 0]} = \text{ATE} + \text{selection bias} + \underbrace{(1 - p)(\text{ATT} - \text{ATC})}_{\text{heterogeneity bias}}.

Selection bias (selection on levels): the treated would have had different outcomes even untreated.
Heterogeneity bias (selection on gains): the treated benefit more or less than the untreated.

Randomisation sets both to zero (Day 35).

The Roy model

Roy (1951) asked how workers choose occupations. Applied to programmes: unit $i$ chooses treatment if its gain exceeds its cost,

T_i = \mathbf 1\{Y_i(1) - Y_i(0) > c_i\}.

Choice now depends on potential outcomes, the opposite of an experiment. Two typical patterns:

Positive selection on gains: those who gain most enrol, so ATT > ATE > ATC.
Negative selection on levels: programmes target, or attract, people with poor prospects: $E[Y(0) \mid T = 1] < E[Y(0) \mid T = 0]$ , so naive comparisons understate effects (training for the unemployed, hospital care for the sick).

If gains are normal, $\tau \sim N(\mu, \sigma^2)$ , and $T = \mathbf 1\{\tau > 0\}$ , then

\text{ATT} = \mu + \sigma\frac{\phi(\mu/\sigma)}{\Phi(\mu/\sigma)}.

Ashenfelter's dip

Trainees' earnings often fall just before they enrol (they enrol because they lost a job). A before–after comparison then credits the programme with the natural recovery (Ashenfelter, 1978).

Part 1 · Warm-up

A programme has ATT = 2, ATC = 1, 40% of the population treated, and selection bias −3 (all in $1,000 of earnings).

Exercise 1 · ATE

Compute the ATE.

Show solution

Step 1. ATE $= p\cdot\text{ATT} + (1 - p)\cdot\text{ATC}$ (Day 33).

Step 2. $= 0.4(2) + 0.6(1) = 1.4$ .

Answer: $1,400.

Exercise 2 · The naive comparison

What does the naive comparison report?

Show solution

Step 1. Naive $=$ ATT $+$ selection bias $= 2 + (-3)$ .

Answer: −1: the programme looks harmful.

Exercise 3 · Three-term check

Verify the three-term decomposition.

Show solution

Step 1. Heterogeneity bias $= (1 - p)(\text{ATT} - \text{ATC}) = 0.6(2 - 1) = 0.6$ .

Step 2. ATE $+$ SB $+$ HB $= 1.4 - 3 + 0.6 = -1$ . ✓

Exercise 4 · Sign of selection bias

For each, predict the sign of $E[Y(0) \mid T = 1] - E[Y(0) \mid T = 0]$ : (a) earnings, for a programme for displaced workers; (b) health, for people who choose to take vitamins; (c) test scores, for pupils whose parents pay for tutoring.

Show solution

Step 1. (a) Displaced workers would earn less anyway: negative.

Step 2. (b) Vitamin takers tend to be healthier and wealthier: positive.

Step 3. (c) Families who pay for tutoring are richer and more engaged: positive (unless tutoring is bought for struggling pupils: then negative).

Exercise 5 · The Roy rule

Workers enrol if $Y(1) - Y(0) > 1$ . Worker A has $Y(0) = 10$ , $Y(1) = 13$ ; worker B has $Y(0) = 20$ , $Y(1) = 20.5$ . Who enrols?

Show solution

Step 1. A's gain $3 > 1$ : enrols.

Step 2. B's gain $0.5 < 1$ : does not.

Answer: A enrols. Note that A also has lower $Y(0)$ : here selection on gains and negative selection on levels go together.

Part 2 · Proofs

Exercise 6 · The two-term decomposition

Prove naive $=$ ATT $+$ selection bias.

Show solution

Step 1. Observed outcomes: $E[Y \mid T = 1] = E[Y(1) \mid T = 1]$ and $E[Y \mid T = 0] = E[Y(0) \mid T = 0]$ .

Step 2. Add and subtract $E[Y(0) \mid T = 1]$ :

\big(E[Y(1) \mid T = 1] - E[Y(0) \mid T = 1]\big) + \big(E[Y(0) \mid T = 1] - E[Y(0) \mid T = 0]\big).

Step 3. The first bracket is the ATT; the second is the selection bias. $\blacksquare$

Exercise 7 · The three-term decomposition

Prove ATT $=$ ATE $+ (1 - p)(\text{ATT} - \text{ATC})$ , and hence the three-term decomposition.

Show solution

Step 1. From ATE $= p\,\text{ATT} + (1 - p)\,\text{ATC}$ : $\text{ATT} - \text{ATE} = \text{ATT} - p\,\text{ATT} - (1 - p)\,\text{ATC} = (1 - p)(\text{ATT} - \text{ATC})$ .

Step 2. Substitute ATT $=$ ATE $+ (1 - p)(\text{ATT} - \text{ATC})$ into Exercise 6. $\blacksquare$

Exercise 8 · Randomisation kills both biases

Prove that if $T$ is independent of $(Y(0), Y(1))$ , selection bias and heterogeneity bias are both zero.

Show solution

Step 1. Independence of $T$ and $Y(0)$ : $E[Y(0) \mid T = 1] = E[Y(0) \mid T = 0]$ , so selection bias $= 0$ .

Step 2. Independence of $T$ and $\tau = Y(1) - Y(0)$ : ATT $=$ ATC $=$ ATE, so heterogeneity bias $= 0$ .

Step 3. Hence naive $=$ ATE. $\blacksquare$

Exercise 9 · Selection on gains raises the ATT

In the Roy model with $T = \mathbf 1\{\tau > c\}$ (and $0 < P(T = 1) < 1$ ), prove ATT $\ge$ ATE $\ge$ ATC.

Show solution

Step 1. Every treated unit has $\tau > c$ , so ATT $= E[\tau \mid \tau > c] > c$ .

Step 2. Every untreated unit has $\tau \le c$ , so ATC $= E[\tau \mid \tau \le c] \le c$ .

Step 3. So ATT $>$ ATC, and ATE $= p\,\text{ATT} + (1 - p)\,\text{ATC}$ lies between them. $\blacksquare$

Exercise 10 · The Roy ATT with normal gains

If $\tau \sim N(\mu, \sigma^2)$ and $T = \mathbf 1\{\tau > 0\}$ , prove ATT $= \mu + \sigma\phi(\mu/\sigma)/\Phi(\mu/\sigma)$ .

Show solution

Step 1. Standardise: $\tau = \mu + \sigma Z$ , and $\tau > 0 \iff Z > -\mu/\sigma$ .

Step 2. Truncated-normal mean: $E[Z \mid Z > a] = \phi(a)/(1 - \Phi(a))$ with $a = -\mu/\sigma$ .

Step 3. By symmetry, $\phi(-\mu/\sigma) = \phi(\mu/\sigma)$ and $1 - \Phi(-\mu/\sigma) = \Phi(\mu/\sigma)$ .

Step 4. ATT $= \mu + \sigma E[Z \mid Z > a] = \mu + \sigma\phi(\mu/\sigma)/\Phi(\mu/\sigma)$ . $\blacksquare$

Part 3 · Applied

Exercise 11 · Roy model for a training programme

Gains from training (in $1,000) are normally distributed with mean 2 and sd 3 across workers, and workers enrol only if they gain. Compute the share who enrol, the ATT and the ATE.

Show solution

Step 1. Share enrolling: $P(\tau > 0) = \Phi(2/3) = \Phi(0.667) \approx 0.748$ .

Step 2. $\phi(0.667) = 0.3989\,e^{-0.667^2/2} \approx 0.3989 \times 0.801 \approx 0.319$ .

Step 3. ATT $= 2 + 3 \times 0.319/0.748 \approx 2 + 1.28 = 3.28$ .

Step 4. ATE $= \mu = 2$ .

Answer: 75% enrol; ATT ≈ $3,280 versus ATE $2,000. An evaluation of current trainees overstates what compulsory training would achieve.

Exercise 12 · Ashenfelter's dip

Trainees usually earn $15k a year. In the year before training, after losing jobs, they earn $12k. After training they earn $16k. Without training, their earnings would have recovered to $15k. Compare the before–after estimate with the true effect.

Show solution

Step 1. Before–after: $16 - 12 = 4$ .

Step 2. True effect: $Y(1) - Y(0) = 16 - 15 = 1$ .

Step 3. The extra 3 is the recovery from the dip (mean reversion) that would have happened anyway.

Answer: before–after says $4k; the truth is $1k. Comparing with a pre-dip baseline or with a matched control group that also lost jobs avoids this.

Exercise 13 · Do hospitals harm health?

In a health survey, people admitted to hospital last year report worse health (3.2 on a 1–5 scale, 5 = excellent) than those not admitted (3.9). Decompose the naive difference if hospital care truly raises health by 0.3 for those admitted.

Show solution

Step 1. Naive $= 3.2 - 3.9 = -0.7$ .

Step 2. Naive $=$ ATT $+$ selection bias, with ATT $= +0.3$ .

Step 3. Selection bias $= -0.7 - 0.3 = -1.0$ .

Answer: admitted patients would have reported health a full point lower without care; the sick go to hospital. The naive comparison makes beneficial care look harmful.

Exercise 14 · The college wage premium

College graduates earn 60% more than high-school graduates (in logs, 0.47). A twins study suggests ability and family background account for 0.15 log points. Decompose the raw gap.

Show solution

Step 1. Raw gap $= 0.47$ = ATT $+$ selection bias, with selection bias $= 0.15$ (graduates would have earned more anyway).

Step 2. ATT $= 0.47 - 0.15 = 0.32$ , i.e. $e^{0.32} - 1 \approx 38\%$ .

Answer: about two-thirds of the raw premium is a causal return for graduates; one-third is selection. If gains are heterogeneous, the effect for non-graduates (ATC) may be smaller still.

Exercise 15 · Which direction?

A city opens a free clinic, and clinic users have lower blood pressure than non-users. An analyst concludes the clinic works. List the threats using the decomposition, and propose a design that avoids them.

Show solution

Step 1. Selection on levels: health-conscious people use clinics and would have lower blood pressure anyway (positive bias in the naive "effect", i.e. too good).

Step 2. Selection on levels the other way: people with symptoms seek care (negative bias). The net sign is unknown.

Step 3. Selection on gains: those who expect to benefit attend, so even an unbiased ATT would overstate the ATE of universal access.

Step 4. Design: randomise invitations or outreach (an encouragement design, Day 41), or exploit a clinic opening date that varies across neighbourhoods (difference-in-differences, Phase 5).

Answer: the naive comparison mixes the clinic's effect with who chooses to attend; randomised encouragement isolates it.

Verify in Python

import numpy as np
from scipy import stats

# Roy model with normal gains (Exercise 11)
mu, sd = 2.0, 3.0
p = stats.norm.cdf(mu / sd)
att = mu + sd * stats.norm.pdf(mu / sd) / p
print(f"share enrolling {p:.3f}, ATT {att:.3f}, ATE {mu}")

rng = np.random.default_rng(34)
n = 1_000_000
y0 = rng.normal(20, 5, n)
tau = rng.normal(mu, sd, n)
T = tau > 0                                         # select on gains only
y = y0 + T * tau
naive = y[T].mean() - y[~T].mean()
sb = y0[T].mean() - y0[~T].mean()
att_s, atc_s = tau[T].mean(), tau[~T].mean()
print(f"simulated ATT {att_s:.3f}, ATC {atc_s:.3f}, ATE {tau.mean():.3f}")
print(f"naive {naive:.3f} = ATE {tau.mean():.3f} + SB {sb:.3f} + HB {(1 - T.mean()) * (att_s - atc_s):.3f}")

Key takeaways

Naive comparison = ATE + selection bias + heterogeneity bias.
In the Roy model people select on gains, so ATT > ATE > ATC; programmes for the disadvantaged also select on levels.
Before–after comparisons are vulnerable to Ashenfelter's dip and mean reversion.

Day 34. The Fundamental Problem & Selection Bias

Theory

Two decompositions

The Roy model

Ashenfelter's dip

Part 1 · Warm-up

Exercise 1 · ATE

Exercise 2 · The naive comparison

Exercise 3 · Three-term check

Exercise 4 · Sign of selection bias

Exercise 5 · The Roy rule

Part 2 · Proofs

Exercise 6 · The two-term decomposition

Exercise 7 · The three-term decomposition

Exercise 8 · Randomisation kills both biases

Exercise 9 · Selection on gains raises the ATT

Exercise 10 · The Roy ATT with normal gains

Part 3 · Applied

Exercise 11 · Roy model for a training programme

Exercise 12 · Ashenfelter's dip

Exercise 13 · Do hospitals harm health?

Exercise 14 · The college wage premium

Exercise 15 · Which direction?

Verify in Python

Key takeaways

Simulation Lab: The Roy Model of Self-Selection