Day 28: Frisch–Waugh–Lovell & Partialling Out

Theory

The theorem

Split the regressors: $y = X_1\beta_1 + X_2\beta_2 + \varepsilon$ . Let $M_2 = I - X_2(X_2^{\top}X_2)^{-1}X_2^{\top}$ , the residual-maker for $X_2$ (Day 20). Define the residualised variables

\tilde X_1 = M_2X_1, \qquad \tilde y = M_2y.

FWL theorem. The OLS coefficient on $X_1$ in the full regression equals the coefficient from regressing $\tilde y$ on $\tilde X_1$ :

\hat\beta_1 = (\tilde X_1^{\top}\tilde X_1)^{-1}\tilde X_1^{\top}\tilde y = (\tilde X_1^{\top}\tilde X_1)^{-1}\tilde X_1^{\top}y,

and the residuals of the two regressions are identical.

What it means

"Controlling for $X_2$ " means: strip out of both $y$ and $X_1$ everything linearly predictable from $X_2$ , then relate what is left. For a single regressor this is the regression anatomy formula

\hat\beta_1 = \frac{\widehat{\operatorname{Cov}}(y, \tilde x_1)}{\widehat{\operatorname{Var}}(\tilde x_1)}.

Consequences you will use repeatedly:

Fixed effects: with $X_2$ = group dummies, $M_2$ subtracts group means, so the within (demeaned) regression gives the same slope as including the dummies.
Precision: $\operatorname{Var}(\hat\beta_1) = \sigma^2/\tilde x_1^{\top}\tilde x_1$ , and $\tilde x_1^{\top}\tilde x_1 = (1 - R_1^2)\sum(x_1 - \bar x_1)^2$ , giving the VIF of Day 18.
Added-variable plot: scatter $\tilde y$ against $\tilde x_1$ ; the fitted slope is $\hat\beta_1$ .
Double machine learning (Day 90) replaces the linear projections with flexible machine-learning predictions: residualise $y$ and the treatment on $X$ , then regress residual on residual.

Part 1 · Warm-up

Exercise 1 · What M₂ does

In words, what is $M_2X_1$ ?

Show solution

The residuals from regressing each column of $X_1$ on $X_2$ : the part of $X_1$ that $X_2$ cannot linearly explain.

Exercise 2 · The intercept is partialled out too

Show that the slope in $y = \beta_0 + \beta_1x$ equals the slope from regressing demeaned $y$ on demeaned $x$ without an intercept.

Show solution

Take $X_2 = \mathbf 1$ : $M_2$ is the centring matrix, so $\tilde x = x - \bar x$ and $\tilde y = y - \bar y$ . FWL gives $\hat\beta_1 = \tilde x^{\top}\tilde y/\tilde x^{\top}\tilde x$ , the Cov/Var formula.

Exercise 3 · Partialling out by hand

Four workers: $x_2 = (0, 0, 1, 1)$ marks urban residence, $x_1 = (1, 2, 3, 5)$ is years of post-secondary schooling and $y = (2, 3, 6, 9)$ is hourly earnings (in $10). Regress earnings on an intercept, schooling and the urban dummy. Compute the schooling coefficient $\hat\beta_1$ by FWL.

Show solution

Step 1. Partialling out $(1, x_2)$ means subtracting the rural and urban means (FWL with group dummies).

Step 2. Schooling means: rural $(1 + 2)/2 = 1.5$ , urban $(3 + 5)/2 = 4$ . So $\tilde x_1 = (-0.5, 0.5, -1, 1)$ .

Step 3. Earnings means: rural $2.5$ , urban $7.5$ . So $\tilde y = (-0.5, 0.5, -1.5, 1.5)$ .

Step 4. Residual-on-residual slope:

\hat\beta_1 = \frac{\tilde x_1^{\top}\tilde y}{\tilde x_1^{\top}\tilde x_1} = \frac{0.25 + 0.25 + 1.5 + 1.5}{0.25 + 0.25 + 1 + 1} = \frac{3.5}{2.5} = 1.4.

Answer: each extra year of schooling goes with $14/hour more earnings within the same area.

Exercise 4 · The short regression

For the same data, compute the slope from regressing $y$ on $x_1$ alone (with an intercept).

Show solution

Step 1. Means: $\bar x_1 = 11/4 = 2.75$ and $\bar y = 20/4 = 5$ .

Step 2. Deviations: $(-1.75, -0.75, 0.25, 2.25)$ and $(-3, -2, 1, 4)$ .

Step 3. Cross-product $5.25 + 1.5 + 0.25 + 9 = 16$ ; squared deviations $3.0625 + 0.5625 + 0.0625 + 5.0625 = 8.75$ .

Step 4. Slope $= 16/8.75 \approx 1.83$ .

Answer: 1.83 without the urban control versus 1.40 with it: part of schooling's raw association with earnings is the urban premium.

Exercise 5 · Only one side needs residualising

Show that $\tilde X_1^{\top}y = \tilde X_1^{\top}\tilde y$ .

Show solution

Step 1. $\tilde X_1^{\top}\tilde y = X_1^{\top}M_2^{\top}M_2y = X_1^{\top}M_2y = \tilde X_1^{\top}y$ , since $M_2$ is symmetric and idempotent.

Step 2. So regressing raw $y$ on $\tilde X_1$ gives the same coefficient (but not the same residuals or SEs).

Part 2 · Proofs

Exercise 6 · Proving FWL

Prove the FWL theorem.

Show solution

Step 1. Write the full OLS fit: $y = X_1\hat\beta_1 + X_2\hat\beta_2 + e$ , with $X_1^{\top}e = 0$ and $X_2^{\top}e = 0$ (Day 20).

Step 2. Multiply through by $M_2$ . $M_2X_2 = 0$ , and $M_2e = e$ because $e$ is already orthogonal to $X_2$ ( $P_2e = 0$ ):

M_2y = M_2X_1\hat\beta_1 + e, \quad\text{i.e.}\quad \tilde y = \tilde X_1\hat\beta_1 + e.

Step 3. Pre-multiply by $\tilde X_1^{\top} = X_1^{\top}M_2$ : $\tilde X_1^{\top}\tilde y = \tilde X_1^{\top}\tilde X_1\hat\beta_1 + X_1^{\top}M_2e$ .

Step 4. $X_1^{\top}M_2e = X_1^{\top}e = 0$ (Step 1).

Step 5. So $\hat\beta_1 = (\tilde X_1^{\top}\tilde X_1)^{-1}\tilde X_1^{\top}\tilde y$ . $\blacksquare$

Exercise 7 · Identical residuals

Show that the residuals from regressing $\tilde y$ on $\tilde X_1$ equal the full-regression residuals $e$ .

Show solution

From Exercise 6, $\tilde y - \tilde X_1\hat\beta_1 = e$ . So $e^{\top}e$ is the same, but the residualised regression should still use $n - k$ (all regressors), not $n - k_1$ , when computing $s^2$ .

Exercise 8 · Regression anatomy

For a single regressor $x_1$ , show $\hat\beta_1 = \widehat{\operatorname{Cov}}(y, \tilde x_1)/\widehat{\operatorname{Var}}(\tilde x_1)$ .

Show solution

Step 1. By FWL and Exercise 5, $\hat\beta_1 = \tilde x_1^{\top}y/\tilde x_1^{\top}\tilde x_1$ .

Step 2. If $X_2$ includes an intercept, $\tilde x_1$ has mean zero, so $\tilde x_1^{\top}y = \sum\tilde x_{1i}(y_i - \bar y)$ , i.e. $n - 1$ times the sample covariance; the denominator is $n - 1$ times the variance.

Exercise 9 · Orthogonal controls change nothing

Prove that if $X_1^{\top}X_2 = 0$ , adding $X_2$ leaves $\hat\beta_1$ unchanged.

Show solution

Step 1. $\tilde X_1 = X_1 - X_2(X_2^{\top}X_2)^{-1}X_2^{\top}X_1 = X_1$ .

Step 2. FWL then gives $(X_1^{\top}X_1)^{-1}X_1^{\top}y$ , the short-regression coefficient.

Step 3. Randomised treatments are (approximately) orthogonal to pre-treatment covariates, which is why adding covariates changes the point estimate little while improving precision (Day 27).

Exercise 10 · Fixed effects are FWL

Show that including a full set of group dummies $D$ is equivalent to demeaning $y$ and $X_1$ by group.

Show solution

Step 1. With $X_2 = D$ (no separate intercept), $D^{\top}D = \operatorname{diag}(n_g)$ and $D^{\top}y$ stacks group sums, so $P_Dy$ replaces each $y_i$ by its group mean.

Step 2. Hence $M_Dy = y_i - \bar y_{g(i)}$ and likewise for $X_1$ ; FWL gives the within estimator.

Part 3 · Applied

Exercise 11 · Interpreting the change

In Exercises 3–4 the coefficient on $x_1$ falls from 1.83 to 1.40 when $x_2$ is added. Explain using the OVB formula.

Show solution

Step 1. Urban workers have more schooling (group means 4 versus 1.5).

Step 2. Urban residence raises earnings at a given schooling level (the full regression gives a coefficient of 1.5 on the dummy, i.e. $15/hour).

Step 3. Omitting it loads part of the urban premium onto schooling: OVB = (effect of urban) × (slope of urban on schooling) > 0.

Answer: the drop from 1.83 to 1.40 removes that bias, provided urban residence is a confounder. If schooling caused people to move to cities (a mediator), controlling for it would instead remove part of schooling's effect.

Exercise 12 · Two-way fixed effects

In a balanced panel, show that partialling out unit and time dummies transforms $x_{it}$ into $x_{it} - \bar x_{i\cdot} - \bar x_{\cdot t} + \bar x_{\cdot\cdot}$ .

Show solution

Step 1. Projecting on unit dummies replaces $x_{it}$ by the unit mean $\bar x_{i\cdot}$ ; projecting on time dummies replaces it by the period mean $\bar x_{\cdot t}$ .

Step 2. In a balanced panel the two sets of dummies are orthogonal once the overall mean is removed, so the projection onto both adds the two parts and subtracts the grand mean once (otherwise it would be counted twice): $\bar x_{i\cdot} + \bar x_{\cdot t} - \bar x_{\cdot\cdot}$ .

Step 3. The residual is $x_{it} - \bar x_{i\cdot} - \bar x_{\cdot t} + \bar x_{\cdot\cdot}$ .

Answer: two-way FE = double demeaning. With unbalanced panels or staggered treatment this simple form fails, part of the story of Day 70.

Exercise 13 · OLS weights heterogeneous effects by treatment variance

Two strata of equal size have treatment effects 1 and 3. The treatment probability is 0.5 in stratum A and 0.1 in stratum B. Regressing $y$ on $T$ and stratum dummies gives approximately $\sum_sw_s\tau_s$ with $w_s \propto P(s)\,p_s(1 - p_s)$ . Compute it and compare with the ATE.

Show solution

Step 1. Conditional treatment variances: stratum A $0.5 \times 0.5 = 0.25$ ; stratum B $0.1 \times 0.9 = 0.09$ .

Step 2. Weights (equal stratum sizes): $w_A = 0.25/(0.25 + 0.09) \approx 0.735$ , $w_B \approx 0.265$ .

Step 3. Regression estimate: $0.735(1) + 0.265(3) \approx 1.53$ .

Step 4. ATE: $0.5(1) + 0.5(3) = 2$ .

Answer: 1.53 versus 2. With heterogeneous effects, OLS with controls estimates a variance-weighted average (Angrist, 1998), favouring strata where treatment is closest to 50/50.

Exercise 14 · Partialling out with machine learning: 401(k) eligibility

To estimate the effect of 401(k) eligibility $D$ on household savings $y$ , controlling flexibly for income, age and education $X$ , double ML assumes $y = \theta D + g(X) + \varepsilon$ and regresses $y - \hat E[y \mid X]$ on $D - \hat E[D \mid X]$ , with both predictions from machine learning. Why is this FWL?

Show solution

Step 1. FWL removes from $y$ and $D$ everything linearly predictable from the controls, then regresses residual on residual.

Step 2. Here the linear projection is replaced by the conditional expectation $E[\cdot \mid X]$ , estimated flexibly (e.g. random forests), so $g(X)$ need not be linear.

Step 3. The residualised $D$ is the variation in eligibility not explained by income, age and education; its relation to residualised savings identifies $\theta$ .

Answer: it is FWL with machine-learning projections (Robinson, 1988; Day 90).

Exercise 15 · An added-variable plot

Why can an added-variable plot reveal an influential observation that a raw scatter of $y$ against $x_1$ hides?

Show solution

Step 1. By FWL, $\hat\beta_1$ depends only on the residualised data $(\tilde x_1, \tilde y)$ .

Step 2. A unit can look ordinary in raw $x_1$ yet be unusual given $X_2$ (a large $\tilde x_1$ ), which gives it high leverage on $\hat\beta_1$ .

Step 3. A raw scatter of $y$ on $x_1$ hides this; the added-variable plot of $\tilde y$ against $\tilde x_1$ shows it.

Answer: the added-variable plot shows the data the coefficient actually uses (see the FWL simulator).

Verify in Python

import numpy as np

x2 = np.array([0, 0, 1, 1.]); x1 = np.array([1, 2, 3, 5.]); y = np.array([2, 3, 6, 9.])
X = np.column_stack([np.ones(4), x1, x2])
b_full = np.linalg.lstsq(X, y, rcond=None)[0]
X2 = np.column_stack([np.ones(4), x2]); M2 = np.eye(4) - X2 @ np.linalg.pinv(X2)
x1t, yt = M2 @ x1, M2 @ y
print("full regression:", b_full.round(4))
print("FWL slope:", round(x1t @ yt / (x1t @ x1t), 4), " raw y on residualised x1:", round(x1t @ y / (x1t @ x1t), 4))
print("short regression slope:", round(np.polyfit(x1, y, 1)[0], 4))
e_full = y - X @ b_full; e_fwl = yt - x1t * (x1t @ yt / (x1t @ x1t))
print("same residuals:", np.allclose(e_full, e_fwl))

# Variance-weighted effects (Exercise 13)
rng = np.random.default_rng(28); n = 400_000
s = rng.random(n) < 0.5                                       # stratum B
p = np.where(s, 0.1, 0.5); T = rng.random(n) < p
Y = np.where(s, 5, 0) + np.where(s, 3, 1) * T + rng.normal(size=n)
Z = np.column_stack([np.ones(n), T, s])
print("regression estimate:", np.linalg.lstsq(Z, Y, rcond=None)[0][1].round(3), " ATE: 2")

Key takeaways

A multiple-regression coefficient is a simple regression of residualised $y$ on residualised $x$ .
Fixed effects, demeaning and "controlling for" are all FWL.
With heterogeneous effects, OLS with controls weights strata by the conditional variance of treatment, not by population share.

Day 28. Frisch–Waugh–Lovell & Partialling Out

Theory

The theorem

What it means

Part 1 · Warm-up

Exercise 1 · What M₂ does

Exercise 2 · The intercept is partialled out too

Exercise 3 · Partialling out by hand

Exercise 4 · The short regression

Exercise 5 · Only one side needs residualising

Part 2 · Proofs

Exercise 6 · Proving FWL

Exercise 7 · Identical residuals

Exercise 8 · Regression anatomy

Exercise 9 · Orthogonal controls change nothing

Exercise 10 · Fixed effects are FWL

Part 3 · Applied

Exercise 11 · Interpreting the change

Exercise 12 · Two-way fixed effects

Exercise 13 · OLS weights heterogeneous effects by treatment variance

Exercise 14 · Partialling out with machine learning: 401(k) eligibility

Exercise 15 · An added-variable plot

Verify in Python

Key takeaways

Simulation Lab: Partialling Out (FWL)