{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 20: Projections, Orthogonality & the Geometry of Least Squares\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Least squares is an orthogonal projection. Fitted values, residuals, R² and degrees of freedom all follow from that picture.\n", "\n", "**Learning outcomes**\n", "\n", "- Project a vector onto a line and onto the column space of X.\n", "- Prove that the projection matrices P and M are symmetric and idempotent, with MX = 0.\n", "- Show that OLS residuals are orthogonal to every regressor and derive R² from Pythagoras.\n", "- Compute leverage and explain why adding regressors never lowers R²." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Projection onto a line\n", "\n", "The closest point to $y$ on the line through $u$ is\n", "\n", "$$\n", "\\operatorname{proj}_u y = \\frac{u^{\\top}y}{u^{\\top}u}\\,u,\n", "$$\n", "\n", "and the residual $y - \\operatorname{proj}_u y$ is orthogonal to $u$. The coefficient $u^{\\top}y/u^{\\top}u$ is a regression slope without intercept.\n", "\n", "### Projection onto a column space\n", "\n", "For $X$ with full column rank, define\n", "\n", "$$\n", "P = X(X^{\\top}X)^{-1}X^{\\top}, \\qquad M = I - P.\n", "$$\n", "\n", "- $Py = X\\hat\\beta = \\hat y$ is the point in the column space of $X$ closest to $y$.\n", "- $My = y - \\hat y = e$ is the residual vector.\n", "- $P$ and $M$ are **symmetric** and **idempotent** ($P^2 = P$): projecting twice changes nothing.\n", "- $PX = X$, $MX = 0$, $PM = 0$.\n", "- $\\operatorname{tr}(P) = k$ and $\\operatorname{tr}(M) = n - k$: the residuals live in an $(n - k)$-dimensional space, the origin of \"degrees of freedom\".\n", "\n", "### The picture\n", "\n", "$$\n", "y = \\underbrace{Py}_{\\hat y} + \\underbrace{My}_{e}, \\qquad \\hat y^{\\top}e = 0, \\qquad \\|y\\|^2 = \\|\\hat y\\|^2 + \\|e\\|^2.\n", "$$\n", "\n", "With an intercept, apply the same to centred vectors: $\\text{TSS} = \\text{ESS} + \\text{RSS}$, and\n", "\n", "$$\n", "R^2 = 1 - \\frac{\\text{RSS}}{\\text{TSS}} = \\frac{\\text{ESS}}{\\text{TSS}},\n", "$$\n", "\n", "the squared cosine of the angle between the centred $y$ and the centred $\\hat y$.\n", "\n", "### Leverage\n", "\n", "The diagonal entries $h_{ii}$ of $P$ are **leverages**: how much observation $i$ pulls its own fitted value. They lie in $[0, 1]$ and sum to $k$. In simple regression, $h_{ii} = \\frac1n + \\frac{(x_i - \\bar x)^2}{\\sum_j(x_j - \\bar x)^2}$: points far from $\\bar x$ have high leverage.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Projecting onto an axis\n", "\n", "Project $y = (3, 4)$ onto $u = (1, 0)$ and give the residual.\n", "\n", "
Show solution\n", "\n", "$\\frac{3}{1}(1, 0) = (3, 0)$; residual $(0, 4)$, orthogonal to $u$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Projecting onto a diagonal\n", "\n", "Project $y = (2, 3)$ onto $u = (1, 1)$, and check orthogonality.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Coefficient: $u^{\\top}y/u^{\\top}u = (2 + 3)/(1 + 1) = 2.5$.\n", "\n", "**Step 2.** Projection: $2.5(1, 1) = (2.5, 2.5)$.\n", "\n", "**Step 3.** Residual: $(2, 3) - (2.5, 2.5) = (-0.5, 0.5)$.\n", "\n", "**Step 4.** Orthogonality: $(-0.5)(1) + (0.5)(1) = 0$. ✓\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Projecting onto the constant\n", "\n", "Write $P$ for $X = \\mathbf 1$ ($n = 3$) and compute $Py$ for $y = (1, 2, 6)$.\n", "\n", "
Show solution\n", "\n", "$P = \\mathbf 1(\\mathbf 1^{\\top}\\mathbf 1)^{-1}\\mathbf 1^{\\top} = \\frac13\\mathbf 1\\mathbf 1^{\\top}$, so $Py = \\bar y\\mathbf 1 = (3, 3, 3)$. Regressing on a constant fits the mean.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · The residual-maker\n", "\n", "For the same data, compute $My$.\n", "\n", "
Show solution\n", "\n", "$My = y - \\bar y\\mathbf 1 = (-2, -1, 3)$, the centring matrix of Day 17.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Pythagoras in data\n", "\n", "Verify $\\|y\\|^2 = \\|Py\\|^2 + \\|My\\|^2$ for Exercise 3.\n", "\n", "
Show solution\n", "\n", "$1 + 4 + 36 = 41$ and $27 + 14 = 41$. ✓\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · P is a projection\n", "\n", "Prove $P^{\\top} = P$ and $P^2 = P$.\n", "\n", "
Show solution\n", "\n", "**Step 1 (symmetry).** Transpose the product in reverse order:\n", "\n", "$$\n", "P^{\\top} = \\big(X(X^{\\top}X)^{-1}X^{\\top}\\big)^{\\top} = X\\big((X^{\\top}X)^{-1}\\big)^{\\top}X^{\\top}.\n", "$$\n", "\n", "**Step 2.** $(X^{\\top}X)^{-1}$ is symmetric (Day 19, Exercise 7), so $P^{\\top} = P$.\n", "\n", "**Step 3 (idempotence).**\n", "\n", "$$\n", "P^2 = X(X^{\\top}X)^{-1}\\underbrace{X^{\\top}X(X^{\\top}X)^{-1}}_{=I}X^{\\top} = X(X^{\\top}X)^{-1}X^{\\top} = P.\n", "$$\n", "\n", "**Step 4.** For $M = I - P$: $M^{\\top} = M$ and $M^2 = I - 2P + P^2 = I - P = M$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · M annihilates X\n", "\n", "Prove $PX = X$, $MX = 0$ and $PM = 0$.\n", "\n", "
Show solution\n", "\n", "$PX = X(X^{\\top}X)^{-1}X^{\\top}X = X$, so $MX = X - X = 0$. $PM = P - P^2 = 0$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Residuals are orthogonal to the regressors\n", "\n", "Show $X^{\\top}e = 0$. With an intercept, deduce that residuals sum to zero and are uncorrelated with each regressor and with $\\hat y$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $X^{\\top}e = X^{\\top}My = (MX)^{\\top}y = 0$.\n", "\n", "**Step 2.** The intercept row gives $\\sum e_i = 0$; row $j$ gives $\\sum x_{ij}e_i = 0$, so the sample covariance is zero.\n", "\n", "**Step 3.** Finally $\\hat y^{\\top}e = \\hat\\beta^{\\top}X^{\\top}e = 0$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Degrees of freedom\n", "\n", "Prove $\\operatorname{tr}(P) = k$ and $\\operatorname{tr}(M) = n - k$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** By the cyclic trace (Day 17), $\\operatorname{tr}(P) = \\operatorname{tr}\\big((X^{\\top}X)^{-1}X^{\\top}X\\big) = \\operatorname{tr}(I_k) = k$, and $\\operatorname{tr}(M) = \\operatorname{tr}(I_n) - k = n - k$.\n", "\n", "**Step 2.** On Day 27 this gives $E[e^{\\top}e] = \\sigma^2(n - k)$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Least squares really is least\n", "\n", "Show that for any $b$, $\\|y - Xb\\|^2 = \\|y - X\\hat\\beta\\|^2 + \\|X(\\hat\\beta - b)\\|^2$, so $\\hat\\beta$ minimises the sum of squared residuals.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $y - Xb = e + X(\\hat\\beta - b)$.\n", "\n", "**Step 2.** The cross term is $2(\\hat\\beta - b)^{\\top}X^{\\top}e = 0$ by Exercise 8, so Pythagoras applies.\n", "\n", "**Step 3.** The second term is $\\ge 0$, and zero only when $X(\\hat\\beta - b) = 0$, i.e. $b = \\hat\\beta$ under full rank.\n", "\n", "
\n", "\n", "## Part 3 · Applied\n", "\n", "Use the four farms from Day 17: advisory visits $x = (1, 2, 3, 4)$ and maize yields $y = (2, 3, 5, 6)$ t/ha, with an intercept. OLS gives $\\hat\\beta = (0.5, 1.4)$: 1.4 extra tonnes per hectare per visit." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Orthogonality in numbers\n", "\n", "The residuals are $e = (0.1, -0.3, 0.3, -0.1)$. Check $X^{\\top}e = 0$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Intercept row of $X^{\\top}e$: $\\sum e_i = 0.1 - 0.3 + 0.3 - 0.1 = 0$.\n", "\n", "**Step 2.** Visits row: $\\sum x_ie_i = 1(0.1) + 2(-0.3) + 3(0.3) + 4(-0.1) = 0.1 - 0.6 + 0.9 - 0.4 = 0$.\n", "\n", "**Answer:** $X^{\\top}e = 0$: the yield residuals are uncorrelated with the number of visits.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · R²\n", "\n", "Compute TSS, RSS and $R^2$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Mean yield $\\bar y = 16/4 = 4$.\n", "\n", "**Step 2.** TSS $= \\sum(y_i - 4)^2 = 4 + 1 + 1 + 4 = 10$.\n", "\n", "**Step 3.** RSS $= \\sum e_i^2 = 0.01 + 0.09 + 0.09 + 0.01 = 0.2$.\n", "\n", "**Step 4.** $R^2 = 1 - 0.2/10 = 0.98$.\n", "\n", "**Answer:** visits account for 98% of the variation in yields across these four farms.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Leverage\n", "\n", "Compute the leverage of each observation and check that they sum to $k = 2$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Simple-regression leverage: $h_{ii} = \\frac1n + \\frac{(x_i - \\bar x)^2}{\\sum_j(x_j - \\bar x)^2}$, with $\\bar x = 2.5$ and $\\sum(x - \\bar x)^2 = 2.25 + 0.25 + 0.25 + 2.25 = 5$.\n", "\n", "**Step 2.** Farms with 1 or 4 visits: $0.25 + 2.25/5 = 0.70$.\n", "\n", "**Step 3.** Farms with 2 or 3 visits: $0.25 + 0.25/5 = 0.30$.\n", "\n", "**Step 4.** Sum: $0.7 + 0.3 + 0.3 + 0.7 = 2 = k$. ✓\n", "\n", "**Answer:** the two extreme farms each pull their own fitted yield with leverage 0.70, more than twice the middle farms.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Regression on a training dummy fits group means\n", "\n", "Two trained workers earn $(5, 7)$ and two untrained workers earn $(1, 3)$ (in \\$10/hour). What are the fitted values from regressing earnings on an intercept and the training dummy $T$?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The column space of $(\\mathbf 1, T)$ contains exactly the vectors that are constant within each group.\n", "\n", "**Step 2.** The closest such vector to $y$ replaces each value by its group mean: trained $(5 + 7)/2 = 6$; untrained $(1 + 3)/2 = 2$.\n", "\n", "**Answer:** $\\hat y = (6, 6, 2, 2)$; the slope is $6 - 2 = 4$, the difference in mean earnings.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Adding a meaningless control never lowers R²\n", "\n", "An analyst adds the last digit of each worker's postcode to a wage regression and $R^2$ rises slightly. Prove that adding any regressor cannot lower $R^2$, and explain what this implies for choosing controls.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The column space of $[X, z]$ contains that of $X$.\n", "\n", "**Step 2.** OLS fitted values are the closest point to $y$ in the column space. Searching a larger set cannot give a point further away, so RSS can only fall or stay equal.\n", "\n", "**Step 3.** TSS does not depend on the regressors, so $R^2 = 1 - \\text{RSS}/\\text{TSS}$ can only rise or stay equal.\n", "\n", "**Answer:** even the postcode digit raises $R^2$ a little by chance. $R^2$ is therefore a poor guide to whether a control belongs in a causal regression; that is decided by the causal structure (Phase 4).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "x = np.array([1, 2, 3, 4.]); y = np.array([2, 3, 5, 6.])\n", "X = np.column_stack([np.ones(4), x])\n", "P = X @ np.linalg.inv(X.T @ X) @ X.T; M = np.eye(4) - P\n", "print(\"symmetric:\", np.allclose(P, P.T), \" idempotent:\", np.allclose(P @ P, P), \" MX = 0:\", np.allclose(M @ X, 0))\n", "print(\"trace P, trace M:\", round(np.trace(P), 6), round(np.trace(M), 6))\n", "e = M @ y; yhat = P @ y\n", "print(\"fitted:\", yhat.round(2), \" residuals:\", e.round(2), \" X'e:\", (X.T @ e).round(10))\n", "tss = ((y - y.mean())**2).sum(); rss = e @ e\n", "print(\"R2 =\", 1 - rss / tss, \" leverages:\", np.diag(P).round(2))\n", "\n", "# Adding a pure-noise regressor never lowers R2\n", "z = np.random.default_rng(20).normal(size=4)\n", "X2 = np.column_stack([X, z]); e2 = y - X2 @ np.linalg.lstsq(X2, y, rcond=None)[0]\n", "print(\"R2 with noise regressor:\", round(1 - e2 @ e2 / tss, 4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- OLS fitted values are the orthogonal projection of $y$ onto the column space of $X$; residuals are what is left, perpendicular to every regressor.\n", "- $P$ and $M$ are symmetric and idempotent; their traces are the model and residual degrees of freedom.\n", "- $R^2$ comes from Pythagoras and can only rise when regressors are added." ] } ] }