Day 39: Regression Adjustment in Experiments (Lin's Estimator)

Theory

Why adjust?

Randomisation already makes the difference in means unbiased. Adjusting for a baseline covariate $X$ that predicts the outcome removes outcome noise (Day 27, Exercise 14) and corrects chance imbalance (Day 35, Exercise 9).

ANCOVA

Regress $Y$ on an intercept, $T$ and $X$ :

\hat\tau_{\text{ANCOVA}} = (\bar Y_1 - \bar Y_0) - \hat\beta(\bar X_1 - \bar X_0),

where $\hat\beta$ is the pooled within-arm slope. The raw difference is corrected for the covariate imbalance.

Freedman's critique

Freedman (2008) showed that, under randomisation alone, ANCOVA is biased in small samples (bias of order $1/n$ ) and can be less precise than the raw difference when treatment effects vary with $X$ and the arms have unequal sizes.

Lin's estimator

Lin (2013) answered: interact treatment with the centred covariate,

Y = \alpha + \tau T + \beta(X - \bar X) + \gamma\,T(X - \bar X) + e,

and use heteroskedasticity-robust SEs. Equivalently, fit separate regressions in each arm (slopes $\hat\beta_1, \hat\beta_0$ ) and compare predictions at the overall mean $\bar X$ :

\hat\tau_{\text{Lin}} = \big[\bar Y_1 - \hat\beta_1(\bar X_1 - \bar X)\big] - \big[\bar Y_0 - \hat\beta_0(\bar X_0 - \bar X)\big].

Properties: consistent; asymptotically at least as precise as the difference in means and as ANCOVA; robust SEs are (asymptotically) conservative. With a categorical $X$ , it is exactly the post-stratified estimator (Day 38).

Good practice

Pre-specify a small number of strong baseline predictors (above all, the baseline value of the outcome). Never choose covariates after looking at the estimated effects.

Part 1 · Warm-up

Exercise 1 · ANCOVA by hand

In a job-training RCT, treated workers earn $1,200 more at follow-up, but they also earned $500 more at baseline. The pooled within-arm slope of follow-up on baseline earnings is 0.8. Compute the ANCOVA estimate.

Show solution

Step 1. Correction: $\hat\beta(\bar X_1 - \bar X_0) = 0.8 \times 500 = 400$ .

Step 2. $\hat\tau_{\text{ANCOVA}} = 1{,}200 - 400 = 800$ .

Answer: $800: one-third of the raw gap reflected the baseline head start.

Exercise 2 · Lin by hand

Test scores: $\bar Y_1 = 20$ , $\bar Y_0 = 17$ ; baseline $\bar X_1 = 10.5$ , $\bar X_0 = 9.5$ , $\bar X = 10$ ; within-arm slopes $\hat\beta_1 = 1.2$ , $\hat\beta_0 = 0.8$ . Compute $\hat\tau_{\text{Lin}}$ and compare with the raw difference.

Show solution

Step 1. Treated prediction at $\bar X$ : $20 - 1.2(10.5 - 10) = 20 - 0.6 = 19.4$ .

Step 2. Control prediction at $\bar X$ : $17 - 0.8(9.5 - 10) = 17 + 0.4 = 17.4$ .

Step 3. $\hat\tau_{\text{Lin}} = 19.4 - 17.4 = 2.0$ , versus the raw difference 3.

Answer: 2.0. The treated group started ahead, and part of the raw gap was that head start.

Exercise 3 · Precision gain

The baseline score explains 50% of the variance of end-line scores within each arm. Approximately how much does adjustment shrink the SE?

Show solution

Step 1. Residual variance falls to $(1 - 0.5)\sigma^2$ .

Step 2. SE falls by $\sqrt{0.5} \approx 0.71$ .

Answer: about 29% smaller, worth doubling the sample (Day 27).

Exercise 4 · Lin with a binary covariate

Urban (60% of households) and rural (40%). Effects of a cash transfer estimated within each: urban 10, rural 25. What does Lin's estimator with an urban dummy give?

Show solution

Step 1. With a binary $X$ , Lin's estimator is the post-stratified estimator (Exercise 8).

Step 2. $0.6(10) + 0.4(25) = 6 + 10 = 16$ .

Answer: 16.

Exercise 5 · Which covariates?

An RCT of a reading programme measures: baseline reading score, age, gender, classroom noise during the programme, and end-line attendance. Which should go in the adjustment?

Show solution

Step 1. Baseline reading score: strongest predictor, measured before randomisation: yes.

Step 2. Age and gender: pre-treatment, modestly predictive: acceptable if pre-specified.

Step 3. Noise during the programme and end-line attendance: measured after assignment, possibly affected by treatment: no.

Answer: baseline score (plus age and gender if pre-specified).

Part 2 · Proofs

Exercise 6 · The ANCOVA formula

Prove $\hat\tau_{\text{ANCOVA}} = (\bar Y_1 - \bar Y_0) - \hat\beta(\bar X_1 - \bar X_0)$ .

Show solution

Step 1. The regressors include an intercept and $T$ , whose span contains both arm indicators. Residuals are orthogonal to them, so they average to zero within each arm.

Step 2. Averaging the fitted equation within the treated arm: $\bar Y_1 = \hat\alpha + \hat\tau + \hat\beta\bar X_1$ .

Step 3. Within the control arm: $\bar Y_0 = \hat\alpha + \hat\beta\bar X_0$ .

Step 4. Subtract: $\bar Y_1 - \bar Y_0 = \hat\tau + \hat\beta(\bar X_1 - \bar X_0)$ . Rearrange. $\blacksquare$

Exercise 7 · Lin = separate regressions

Show that the interacted regression gives the same fitted values as separate regressions of $Y$ on $X$ in each arm, and that $\hat\tau$ is the difference of the two fitted lines at $\bar X$ .

Show solution

Step 1. The four regressors $1, T, (X - \bar X), T(X - \bar X)$ span the same space as $(1 - T), T, (1 - T)(X - \bar X), T(X - \bar X)$ : an intercept and slope for each arm.

Step 2. So OLS fits each arm separately: control line $\hat\alpha + \hat\beta(X - \bar X)$ , treated line $\hat\alpha + \hat\tau + (\hat\beta + \hat\gamma)(X - \bar X)$ .

Step 3. At $X = \bar X$ , the lines are $\hat\alpha$ and $\hat\alpha + \hat\tau$ , so $\hat\tau$ is their difference.

Step 4. Each arm's line passes through its means, so its value at $\bar X$ is $\bar Y_t - \hat\beta_t(\bar X_t - \bar X)$ . $\blacksquare$

Exercise 8 · Lin = post-stratification for a categorical covariate

With $X$ a set of category dummies (centred), show $\hat\tau_{\text{Lin}} = \sum_g(N_g/N)\hat\tau_g$ .

Show solution

Step 1. Separate regressions on category dummies in each arm fit the category-by-arm means $\bar Y_{tg}$ (Day 20, Exercise 14).

Step 2. "Evaluating at the overall mean of the centred dummies" means averaging category predictions with weights $N_g/N$ .

Step 3. So $\hat\tau_{\text{Lin}} = \sum_g\frac{N_g}{N}\bar Y_{1g} - \sum_g\frac{N_g}{N}\bar Y_{0g} = \sum_g\frac{N_g}{N}\hat\tau_g$ . $\blacksquare$

Exercise 9 · Why ANCOVA can hurt

Explain, using Exercise 6, why ANCOVA's pooled slope can make it less precise than the raw difference when the arms are unbalanced (e.g. 80/20) and slopes differ between arms.

Show solution

Step 1. The ideal correction for the treated mean uses $\beta_1$ , and for the control mean uses $\beta_0$ .

Step 2. ANCOVA uses one pooled slope, which with 80% treated is close to $\beta_1$ , but the variance of $\hat\tau$ is dominated by the smaller arm (20% controls), whose correction then uses the wrong slope.

Step 3. With $\beta_1$ and $\beta_0$ far apart, the mis-corrected control mean can be noisier than the unadjusted one.

Answer: separate slopes (Lin) fix this; asymptotically Lin's variance is never larger than the difference in means' (Lin, 2013).

Exercise 10 · Small-sample bias

Lin's estimator is consistent but not exactly unbiased. Why does estimating slopes introduce bias of order $1/n$ , and why does this rarely matter?

Show solution

Step 1. $\hat\beta_t$ is estimated from the same data as $\bar X_t$ and $\bar Y_t$ ; the product $\hat\beta_t(\bar X_t - \bar X)$ has a small correlation-induced bias.

Step 2. Each factor has error of order $1/\sqrt n$ , so the product's bias is of order $1/n$ .

Step 3. The SE is of order $1/\sqrt n$ , so the bias is negligible relative to the SE except in very small experiments.

Answer: in very small trials, prefer the difference in means or randomisation inference with the adjusted statistic (Day 37).

Part 3 · Applied

Exercise 11 · Training and baseline earnings

A training RCT (500 per arm) reports: raw difference $1,000 (SE 600); ANCOVA $850 (SE 420); Lin $840 (SE 410). Which should be the headline, and why are the estimates similar?

Show solution

Step 1. With equal arms and a strong pre-treatment predictor, all three are consistent for the ATE; the point estimates differ only through chance imbalance.

Step 2. Adjustment cuts the SE by about 30%: the raw estimate is not significant ( $z = 1.67$ ), Lin's is ( $z \approx 2.05$ ).

Answer: if pre-specified, report Lin's estimate ($840, SE 410) as the headline and show the raw difference too.

Exercise 12 · Specification searching

A team tries 20 combinations of covariates and reports the one with the smallest p-value. Why is this a problem even though each regression is valid?

Show solution

Step 1. Each specification gives a slightly different estimate; choosing the most significant is a multiple-testing problem (Day 10).

Step 2. The reported p-value ignores the search, so the false-positive rate exceeds 5%.

Answer: pre-register the covariates (or use a pre-specified rule such as the baseline outcome only), and report the raw difference as a robustness check.

Exercise 13 · Too many covariates

A trial with 60 participants adjusts for 25 baseline variables. What can go wrong, and what is a better plan?

Show solution

Step 1. With $k$ close to $n$ , the slopes are noisy, the $1/n$ bias grows, and robust SEs become unreliable.

Step 2. Lin's estimator with interactions doubles the number of parameters, making it worse.

Answer: adjust for 1–3 strong pre-specified predictors (the baseline outcome first), or a pre-registered summary index; or block on them at the design stage (Day 38).

Exercise 14 · Adjustment with a binary outcome

A vaccination-reminder RCT measures uptake (0/1) and adjusts for whether the person was vaccinated last season using a linear probability model. Is the adjusted coefficient still the ATE on uptake?

Show solution

Step 1. Lin's results need no outcome model: the regression is just a device for combining arm means and slopes.

Step 2. With a binary covariate, Lin's estimator equals the post-stratified difference in uptake (Exercise 8), an average of risk differences.

Answer: yes; use robust SEs because a linear probability model is heteroskedastic (Day 29).

Exercise 15 · An unbalanced design

A costly treatment is given to 20% of 1,000 firms; effects on sales grow with firm size (different slopes by arm). In simulation (code below), compare the SDs of the raw difference, ANCOVA and Lin.

Show solution

Step 1. Run many randomisations of the same firms and compute the three estimators each time.

Step 2. The simulation shows Lin's estimator has the smallest SD, the raw difference the largest; ANCOVA sits in between and is clearly worse than Lin.

Answer: with unbalanced arms and heterogeneous slopes, use the interacted (Lin) specification.

Verify in Python

import numpy as np

rng = np.random.default_rng(39)
N, N1 = 1000, 200
x = rng.lognormal(0, 0.6, N); xc = x - x.mean()
y0 = 10 + 2 * xc + rng.normal(0, 2, N)
y1 = y0 + 3 + 6 * xc                                  # effect grows with firm size
ate = (y1 - y0).mean()

def ols(X, y):
    return np.linalg.lstsq(X, y, rcond=None)[0]

raw, anc, lin = [], [], []
for _ in range(4000):
    T = np.zeros(N, bool); T[rng.choice(N, N1, replace=False)] = True
    y = np.where(T, y1, y0)
    raw.append(y[T].mean() - y[~T].mean())
    anc.append(ols(np.column_stack([np.ones(N), T, xc]), y)[1])
    lin.append(ols(np.column_stack([np.ones(N), T, xc, T * xc]), y)[1])
for name, e in (("raw difference", raw), ("ANCOVA", anc), ("Lin", lin)):
    print(f"{name:15s} mean {np.mean(e):.3f} (ATE {ate:.3f})  SD {np.std(e):.3f}")

print("Exercise 2 (Lin by hand):", (20 - 1.2 * 0.5) - (17 - 0.8 * (-0.5)))

Key takeaways

Adjusting for strong baseline predictors corrects chance imbalance and cuts the SE.
ANCOVA can misbehave with unbalanced arms and heterogeneous effects; Lin's interacted regression with robust SEs is safe.
Pre-specify a few covariates; never search for the most significant specification.

Day 39. Regression Adjustment in Experiments (Lin's Estimator)

Theory

Why adjust?

ANCOVA

Freedman's critique

Lin's estimator

Good practice

Part 1 · Warm-up

Exercise 1 · ANCOVA by hand

Exercise 2 · Lin by hand

Exercise 3 · Precision gain

Exercise 4 · Lin with a binary covariate

Exercise 5 · Which covariates?

Part 2 · Proofs

Exercise 6 · The ANCOVA formula

Exercise 7 · Lin = separate regressions

Exercise 8 · Lin = post-stratification for a categorical covariate

Exercise 9 · Why ANCOVA can hurt

Exercise 10 · Small-sample bias

Part 3 · Applied

Exercise 11 · Training and baseline earnings

Exercise 12 · Specification searching

Exercise 13 · Too many covariates

Exercise 14 · Adjustment with a binary outcome

Exercise 15 · An unbalanced design

Verify in Python

Key takeaways

Simulation Lab: Raw, ANCOVA & Lin Adjustment