Day 20: Projections, Orthogonality & the Geometry of Least Squares

Theory

Projection onto a line

The closest point to $y$ on the line through $u$ is

\operatorname{proj}_u y = \frac{u^{\top}y}{u^{\top}u}\,u,

and the residual $y - \operatorname{proj}_u y$ is orthogonal to $u$ . The coefficient $u^{\top}y/u^{\top}u$ is a regression slope without intercept.

Projection onto a column space

For $X$ with full column rank, define

P = X(X^{\top}X)^{-1}X^{\top}, \qquad M = I - P.

$Py = X\hat\beta = \hat y$ is the point in the column space of $X$ closest to $y$ .
$My = y - \hat y = e$ is the residual vector.
$P$ and $M$ are symmetric and idempotent ( $P^2 = P$ ): projecting twice changes nothing.
$PX = X$ , $MX = 0$ , $PM = 0$ .
$\operatorname{tr}(P) = k$ and $\operatorname{tr}(M) = n - k$ : the residuals live in an $(n - k)$ -dimensional space, the origin of "degrees of freedom".

The picture

y = \underbrace{Py}_{\hat y} + \underbrace{My}_{e}, \qquad \hat y^{\top}e = 0, \qquad \|y\|^2 = \|\hat y\|^2 + \|e\|^2.

With an intercept, apply the same to centred vectors: $\text{TSS} = \text{ESS} + \text{RSS}$ , and

R^2 = 1 - \frac{\text{RSS}}{\text{TSS}} = \frac{\text{ESS}}{\text{TSS}},

the squared cosine of the angle between the centred $y$ and the centred $\hat y$ .

Leverage

The diagonal entries $h_{ii}$ of $P$ are leverages: how much observation $i$ pulls its own fitted value. They lie in $[0, 1]$ and sum to $k$ . In simple regression, $h_{ii} = \frac1n + \frac{(x_i - \bar x)^2}{\sum_j(x_j - \bar x)^2}$ : points far from $\bar x$ have high leverage.

Part 1 · Warm-up

Exercise 1 · Projecting onto an axis

Project $y = (3, 4)$ onto $u = (1, 0)$ and give the residual.

Show solution

$\frac{3}{1}(1, 0) = (3, 0)$ ; residual $(0, 4)$ , orthogonal to $u$ .

Exercise 2 · Projecting onto a diagonal

Project $y = (2, 3)$ onto $u = (1, 1)$ , and check orthogonality.

Show solution

Step 1. Coefficient: $u^{\top}y/u^{\top}u = (2 + 3)/(1 + 1) = 2.5$ .

Step 2. Projection: $2.5(1, 1) = (2.5, 2.5)$ .

Step 3. Residual: $(2, 3) - (2.5, 2.5) = (-0.5, 0.5)$ .

Step 4. Orthogonality: $(-0.5)(1) + (0.5)(1) = 0$ . ✓

Exercise 3 · Projecting onto the constant

Write $P$ for $X = \mathbf 1$ ( $n = 3$ ) and compute $Py$ for $y = (1, 2, 6)$ .

Show solution

$P = \mathbf 1(\mathbf 1^{\top}\mathbf 1)^{-1}\mathbf 1^{\top} = \frac13\mathbf 1\mathbf 1^{\top}$ , so $Py = \bar y\mathbf 1 = (3, 3, 3)$ . Regressing on a constant fits the mean.

Exercise 4 · The residual-maker

For the same data, compute $My$ .

Show solution

$My = y - \bar y\mathbf 1 = (-2, -1, 3)$ , the centring matrix of Day 17.

Exercise 5 · Pythagoras in data

Verify $\|y\|^2 = \|Py\|^2 + \|My\|^2$ for Exercise 3.

Show solution

$1 + 4 + 36 = 41$ and $27 + 14 = 41$ . ✓

Part 2 · Proofs

Exercise 6 · P is a projection

Prove $P^{\top} = P$ and $P^2 = P$ .

Show solution

Step 1 (symmetry). Transpose the product in reverse order:

P^{\top} = \big(X(X^{\top}X)^{-1}X^{\top}\big)^{\top} = X\big((X^{\top}X)^{-1}\big)^{\top}X^{\top}.

Step 2. $(X^{\top}X)^{-1}$ is symmetric (Day 19, Exercise 7), so $P^{\top} = P$ .

Step 3 (idempotence).

P^2 = X(X^{\top}X)^{-1}\underbrace{X^{\top}X(X^{\top}X)^{-1}}_{=I}X^{\top} = X(X^{\top}X)^{-1}X^{\top} = P.

Step 4. For $M = I - P$ : $M^{\top} = M$ and $M^2 = I - 2P + P^2 = I - P = M$ . $\blacksquare$

Exercise 7 · M annihilates X

Prove $PX = X$ , $MX = 0$ and $PM = 0$ .

Show solution

$PX = X(X^{\top}X)^{-1}X^{\top}X = X$ , so $MX = X - X = 0$ . $PM = P - P^2 = 0$ .

Exercise 8 · Residuals are orthogonal to the regressors

Show $X^{\top}e = 0$ . With an intercept, deduce that residuals sum to zero and are uncorrelated with each regressor and with $\hat y$ .

Show solution

Step 1. $X^{\top}e = X^{\top}My = (MX)^{\top}y = 0$ .

Step 2. The intercept row gives $\sum e_i = 0$ ; row $j$ gives $\sum x_{ij}e_i = 0$ , so the sample covariance is zero.

Step 3. Finally $\hat y^{\top}e = \hat\beta^{\top}X^{\top}e = 0$ .

Exercise 9 · Degrees of freedom

Prove $\operatorname{tr}(P) = k$ and $\operatorname{tr}(M) = n - k$ .

Show solution

Step 1. By the cyclic trace (Day 17), $\operatorname{tr}(P) = \operatorname{tr}\big((X^{\top}X)^{-1}X^{\top}X\big) = \operatorname{tr}(I_k) = k$ , and $\operatorname{tr}(M) = \operatorname{tr}(I_n) - k = n - k$ .

Step 2. On Day 27 this gives $E[e^{\top}e] = \sigma^2(n - k)$ .

Exercise 10 · Least squares really is least

Show that for any $b$ , $\|y - Xb\|^2 = \|y - X\hat\beta\|^2 + \|X(\hat\beta - b)\|^2$ , so $\hat\beta$ minimises the sum of squared residuals.

Show solution

Step 1. $y - Xb = e + X(\hat\beta - b)$ .

Step 2. The cross term is $2(\hat\beta - b)^{\top}X^{\top}e = 0$ by Exercise 8, so Pythagoras applies.

Step 3. The second term is $\ge 0$ , and zero only when $X(\hat\beta - b) = 0$ , i.e. $b = \hat\beta$ under full rank.

Part 3 · Applied

Use the four farms from Day 17: advisory visits $x = (1, 2, 3, 4)$ and maize yields $y = (2, 3, 5, 6)$ t/ha, with an intercept. OLS gives $\hat\beta = (0.5, 1.4)$ : 1.4 extra tonnes per hectare per visit.

Exercise 11 · Orthogonality in numbers

The residuals are $e = (0.1, -0.3, 0.3, -0.1)$ . Check $X^{\top}e = 0$ .

Show solution

Step 1. Intercept row of $X^{\top}e$ : $\sum e_i = 0.1 - 0.3 + 0.3 - 0.1 = 0$ .

Step 2. Visits row: $\sum x_ie_i = 1(0.1) + 2(-0.3) + 3(0.3) + 4(-0.1) = 0.1 - 0.6 + 0.9 - 0.4 = 0$ .

Answer: $X^{\top}e = 0$ : the yield residuals are uncorrelated with the number of visits.

Exercise 12 · R²

Compute TSS, RSS and $R^2$ .

Show solution

Step 1. Mean yield $\bar y = 16/4 = 4$ .

Step 2. TSS $= \sum(y_i - 4)^2 = 4 + 1 + 1 + 4 = 10$ .

Step 3. RSS $= \sum e_i^2 = 0.01 + 0.09 + 0.09 + 0.01 = 0.2$ .

Step 4. $R^2 = 1 - 0.2/10 = 0.98$ .

Answer: visits account for 98% of the variation in yields across these four farms.

Exercise 13 · Leverage

Compute the leverage of each observation and check that they sum to $k = 2$ .

Show solution

Step 1. Simple-regression leverage: $h_{ii} = \frac1n + \frac{(x_i - \bar x)^2}{\sum_j(x_j - \bar x)^2}$ , with $\bar x = 2.5$ and $\sum(x - \bar x)^2 = 2.25 + 0.25 + 0.25 + 2.25 = 5$ .

Step 2. Farms with 1 or 4 visits: $0.25 + 2.25/5 = 0.70$ .

Step 3. Farms with 2 or 3 visits: $0.25 + 0.25/5 = 0.30$ .

Step 4. Sum: $0.7 + 0.3 + 0.3 + 0.7 = 2 = k$ . ✓

Answer: the two extreme farms each pull their own fitted yield with leverage 0.70, more than twice the middle farms.

Exercise 14 · Regression on a training dummy fits group means

Two trained workers earn $(5, 7)$ and two untrained workers earn $(1, 3)$ (in $10/hour). What are the fitted values from regressing earnings on an intercept and the training dummy $T$ ?

Show solution

Step 1. The column space of $(\mathbf 1, T)$ contains exactly the vectors that are constant within each group.

Step 2. The closest such vector to $y$ replaces each value by its group mean: trained $(5 + 7)/2 = 6$ ; untrained $(1 + 3)/2 = 2$ .

Answer: $\hat y = (6, 6, 2, 2)$ ; the slope is $6 - 2 = 4$ , the difference in mean earnings.

Exercise 15 · Adding a meaningless control never lowers R²

An analyst adds the last digit of each worker's postcode to a wage regression and $R^2$ rises slightly. Prove that adding any regressor cannot lower $R^2$ , and explain what this implies for choosing controls.

Show solution

Step 1. The column space of $[X, z]$ contains that of $X$ .

Step 2. OLS fitted values are the closest point to $y$ in the column space. Searching a larger set cannot give a point further away, so RSS can only fall or stay equal.

Step 3. TSS does not depend on the regressors, so $R^2 = 1 - \text{RSS}/\text{TSS}$ can only rise or stay equal.

Answer: even the postcode digit raises $R^2$ a little by chance. $R^2$ is therefore a poor guide to whether a control belongs in a causal regression; that is decided by the causal structure (Phase 4).

Verify in Python

import numpy as np

x = np.array([1, 2, 3, 4.]); y = np.array([2, 3, 5, 6.])
X = np.column_stack([np.ones(4), x])
P = X @ np.linalg.inv(X.T @ X) @ X.T; M = np.eye(4) - P
print("symmetric:", np.allclose(P, P.T), " idempotent:", np.allclose(P @ P, P), " MX = 0:", np.allclose(M @ X, 0))
print("trace P, trace M:", round(np.trace(P), 6), round(np.trace(M), 6))
e = M @ y; yhat = P @ y
print("fitted:", yhat.round(2), " residuals:", e.round(2), " X'e:", (X.T @ e).round(10))
tss = ((y - y.mean())**2).sum(); rss = e @ e
print("R2 =", 1 - rss / tss, " leverages:", np.diag(P).round(2))

# Adding a pure-noise regressor never lowers R2
z = np.random.default_rng(20).normal(size=4)
X2 = np.column_stack([X, z]); e2 = y - X2 @ np.linalg.lstsq(X2, y, rcond=None)[0]
print("R2 with noise regressor:", round(1 - e2 @ e2 / tss, 4))

Key takeaways

OLS fitted values are the orthogonal projection of $y$ onto the column space of $X$ ; residuals are what is left, perpendicular to every regressor.
$P$ and $M$ are symmetric and idempotent; their traces are the model and residual degrees of freedom.
$R^2$ comes from Pythagoras and can only rise when regressors are added.

Day 20. Projections, Orthogonality & the Geometry of Least Squares

Theory

Projection onto a line

Projection onto a column space

The picture

Leverage

Part 1 · Warm-up

Exercise 1 · Projecting onto an axis

Exercise 2 · Projecting onto a diagonal

Exercise 3 · Projecting onto the constant

Exercise 4 · The residual-maker

Exercise 5 · Pythagoras in data

Part 2 · Proofs

Exercise 6 · P is a projection

Exercise 7 · M annihilates X

Exercise 8 · Residuals are orthogonal to the regressors

Exercise 9 · Degrees of freedom

Exercise 10 · Least squares really is least

Part 3 · Applied

Exercise 11 · Orthogonality in numbers

Exercise 12 · R²

Exercise 13 · Leverage

Exercise 14 · Regression on a training dummy fits group means

Exercise 15 · Adding a meaningless control never lowers R²

Verify in Python

Key takeaways

Simulation Lab: Least Squares: Residuals & Orthogonality