{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 21: Eigenvalues, Eigenvectors & Quadratic Forms\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "The directions a matrix only stretches, and what they reveal about variances, collinearity and stability.\n", "\n", "**Learning outcomes**\n", "\n", "- Compute eigenvalues and eigenvectors of 2×2 matrices and check trace and determinant identities.\n", "- Use the spectral decomposition of a symmetric matrix and classify quadratic forms.\n", "- Interpret the eigenvalues of a covariance matrix (principal components) and of X′X (condition number).\n", "- Use eigenvalues to judge the stability of a linear dynamic system." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Definition\n", "\n", "A non-zero vector $v$ is an **eigenvector** of a square matrix $A$, with **eigenvalue** $\\lambda$, if\n", "\n", "$$\n", "Av = \\lambda v.\n", "$$\n", "\n", "$A$ maps $v$ onto its own line, stretching it by $\\lambda$. Eigenvalues solve the **characteristic equation** $\\det(A - \\lambda I) = 0$. For 2×2 matrices,\n", "\n", "$$\n", "\\lambda^2 - \\operatorname{tr}(A)\\,\\lambda + \\det(A) = 0, \\qquad \\lambda_1 + \\lambda_2 = \\operatorname{tr}A, \\qquad \\lambda_1\\lambda_2 = \\det A.\n", "$$\n", "\n", "### Symmetric matrices: the spectral theorem\n", "\n", "If $A = A^{\\top}$ (covariance matrices, $X^{\\top}X$), then all eigenvalues are real, eigenvectors can be chosen orthonormal, and\n", "\n", "$$\n", "A = Q\\Lambda Q^{\\top} = \\sum_j \\lambda_j q_jq_j^{\\top}, \\qquad Q^{\\top}Q = I.\n", "$$\n", "\n", "Consequences: $A^m = Q\\Lambda^mQ^{\\top}$, $A^{-1} = Q\\Lambda^{-1}Q^{\\top}$ (if all $\\lambda_j \\ne 0$), and $A^{1/2} = Q\\Lambda^{1/2}Q^{\\top}$.\n", "\n", "### Quadratic forms and definiteness\n", "\n", "A **quadratic form** is $x^{\\top}Ax$. Writing $x$ in eigen-coordinates $z = Q^{\\top}x$ gives $x^{\\top}Ax = \\sum_j\\lambda_jz_j^2$, so a symmetric $A$ is\n", "\n", "| | eigenvalues | $x^{\\top}Ax$ for $x \\ne 0$ |\n", "|---|---|---|\n", "| positive definite (PD) | all $> 0$ | always $> 0$ |\n", "| positive semi-definite (PSD) | all $\\ge 0$ | always $\\ge 0$ |\n", "| indefinite | mixed signs | both signs |\n", "\n", "The **Rayleigh quotient** $x^{\\top}Ax/x^{\\top}x$ ranges between $\\lambda_{\\min}$ and $\\lambda_{\\max}$.\n", "\n", "### Three uses in econometrics\n", "\n", "- **Principal components:** the eigenvectors of a covariance matrix are the directions of greatest variance; eigenvalues are those variances.\n", "- **Collinearity:** the condition number $\\lambda_{\\max}/\\lambda_{\\min}$ of $X^{\\top}X$ (for standardised regressors) measures how close $X$ is to rank deficiency.\n", "- **Dynamics:** $x_{t+1} = Ax_t$ converges to zero iff every $|\\lambda_j| < 1$.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · A diagonal matrix\n", "\n", "Find the eigenvalues and eigenvectors of $\\begin{pmatrix}2 & 0\\\\0 & 5\\end{pmatrix}$.\n", "\n", "
Show solution\n", "\n", "$\\lambda = 2$ with $v = (1, 0)$ and $\\lambda = 5$ with $v = (0, 1)$: a diagonal matrix stretches along the axes.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · A symmetric matrix\n", "\n", "Find the eigenvalues and eigenvectors of $A = \\begin{pmatrix}2 & 1\\\\1 & 2\\end{pmatrix}$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Characteristic equation: $\\lambda^2 - \\operatorname{tr}(A)\\lambda + \\det A = \\lambda^2 - 4\\lambda + 3 = (\\lambda - 3)(\\lambda - 1) = 0$, so $\\lambda = 3, 1$.\n", "\n", "**Step 2.** For $\\lambda = 3$: $(A - 3I)v = \\begin{pmatrix}-1 & 1\\\\1 & -1\\end{pmatrix}v = 0 \\Rightarrow v_1 = v_2$, e.g. $v = (1, 1)$.\n", "\n", "**Step 3.** For $\\lambda = 1$: $(A - I)v = \\begin{pmatrix}1 & 1\\\\1 & 1\\end{pmatrix}v = 0 \\Rightarrow v_1 = -v_2$, e.g. $v = (1, -1)$.\n", "\n", "**Step 4.** Check orthogonality: $(1, 1)\\cdot(1, -1) = 0$.\n", "\n", "**Answer:** $\\lambda = 3$ along $(1, 1)$ and $\\lambda = 1$ along $(1, -1)$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Trace and determinant\n", "\n", "Check the trace and determinant identities for Exercise 2.\n", "\n", "
Show solution\n", "\n", "$\\operatorname{tr} = 4 = 3 + 1$ and $\\det = 3 = 3 \\times 1$. ✓\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Evaluating a quadratic form\n", "\n", "Compute $x^{\\top}Ax$ for Exercise 2's $A$ with $x = (1, -1)$ and with $x = (1, 1)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $x^{\\top}Ax = 2x_1^2 + 2x_1x_2 + 2x_2^2$.\n", "\n", "**Step 2.** At $(1, -1)$: $2 - 2 + 2 = 2 = 1\\cdot\\|x\\|^2$.\n", "\n", "**Step 3.** At $(1, 1)$: $6 = 3\\cdot\\|x\\|^2$.\n", "\n", "**Step 4.** Along eigenvectors, the form is $\\lambda\\|x\\|^2$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Classifying a form\n", "\n", "Write $3x^2 + 4xy + y^2$ as $x^{\\top}Ax$ and classify it.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $A = \\begin{pmatrix}3 & 2\\\\2 & 1\\end{pmatrix}$ (split the cross term evenly).\n", "\n", "**Step 2.** $\\det A = 3 - 4 = -1 < 0$, so the eigenvalues have opposite signs: **indefinite** (e.g. $x = (1, -2)$ gives $3 - 8 + 4 = -1$).\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Orthogonal eigenvectors\n", "\n", "Prove that eigenvectors of a symmetric matrix belonging to distinct eigenvalues are orthogonal.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Let $Av_1 = \\lambda_1v_1$, $Av_2 = \\lambda_2v_2$.\n", "\n", "**Step 2.** Then $\\lambda_1v_2^{\\top}v_1 = v_2^{\\top}Av_1 = (Av_2)^{\\top}v_1 = \\lambda_2v_2^{\\top}v_1$, using $A = A^{\\top}$.\n", "\n", "**Step 3.** So $(\\lambda_1 - \\lambda_2)v_2^{\\top}v_1 = 0$, and $\\lambda_1 \\ne \\lambda_2$ forces $v_2^{\\top}v_1 = 0$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Trace and determinant from the roots\n", "\n", "For a 2×2 matrix, show $\\lambda_1 + \\lambda_2 = \\operatorname{tr}A$ and $\\lambda_1\\lambda_2 = \\det A$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\det(A - \\lambda I) = (a - \\lambda)(d - \\lambda) - bc = \\lambda^2 - (a + d)\\lambda + (ad - bc)$.\n", "\n", "**Step 2.** A monic quadratic with roots $\\lambda_1, \\lambda_2$ equals $\\lambda^2 - (\\lambda_1 + \\lambda_2)\\lambda + \\lambda_1\\lambda_2$.\n", "\n", "**Step 3.** Match coefficients.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Powers and inverses\n", "\n", "Show that if $Av = \\lambda v$ then $A^mv = \\lambda^mv$, and, if $A$ is invertible, $A^{-1}v = \\lambda^{-1}v$.\n", "\n", "
Show solution\n", "\n", "$A^2v = A(\\lambda v) = \\lambda^2v$, and by induction $A^mv = \\lambda^mv$. Applying $A^{-1}$ to $Av = \\lambda v$ gives $v = \\lambda A^{-1}v$; invertibility forces $\\lambda \\ne 0$, so $A^{-1}v = \\lambda^{-1}v$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Positive definite ⇔ positive eigenvalues\n", "\n", "Using $A = Q\\Lambda Q^{\\top}$, prove that a symmetric matrix is PD iff all its eigenvalues are positive.\n", "\n", "
Show solution\n", "\n", "**Step 1.** With $z = Q^{\\top}x$ (and $z \\ne 0 \\iff x \\ne 0$ since $Q$ is invertible), $x^{\\top}Ax = z^{\\top}\\Lambda z = \\sum_j\\lambda_jz_j^2$.\n", "\n", "**Step 2.** If every $\\lambda_j > 0$ this is positive for $z \\ne 0$.\n", "\n", "**Step 3.** Conversely, taking $x = q_j$ gives $x^{\\top}Ax = \\lambda_j$, which must be positive.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · The Rayleigh quotient\n", "\n", "Prove that $\\max_{\\|x\\| = 1}x^{\\top}Ax = \\lambda_{\\max}$ for symmetric $A$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\|x\\| = 1 \\iff \\|z\\| = 1$ (since $Q$ preserves length).\n", "\n", "**Step 2.** Then $x^{\\top}Ax = \\sum_j\\lambda_jz_j^2 \\le \\lambda_{\\max}\\sum_jz_j^2 = \\lambda_{\\max}$, with equality at $x = q_{\\max}$.\n", "\n", "**Step 3.** This is the first principal component: the unit direction of greatest variance.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Principal components\n", "\n", "Two test scores have covariance matrix $\\Sigma = \\begin{pmatrix}4 & 2\\\\2 & 3\\end{pmatrix}$. Find the eigenvalues and the share of total variance captured by the first principal component.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{tr}\\Sigma = 7$ and $\\det\\Sigma = 4(3) - 2^2 = 8$, so $\\lambda^2 - 7\\lambda + 8 = 0$.\n", "\n", "**Step 2.** $\\lambda = \\frac{7 \\pm \\sqrt{49 - 32}}{2} = \\frac{7 \\pm \\sqrt{17}}{2} \\approx 5.56,\\; 1.44$.\n", "\n", "**Step 3.** Total variance is the trace, 7, so the first component's share is $5.56/7 \\approx 79.5\\%$.\n", "\n", "**Answer:** one \"ability\" index (the first principal component) summarises about 80% of the variation in the two scores.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Stability of a dynamic system\n", "\n", "Employment shares evolve as $x_{t+1} = Ax_t$ with $A = \\begin{pmatrix}0.5 & 0.2\\\\0.1 & 0.6\\end{pmatrix}$ (deviations from steady state). Do deviations die out?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{tr}A = 0.5 + 0.6 = 1.1$ and $\\det A = 0.5(0.6) - 0.2(0.1) = 0.28$.\n", "\n", "**Step 2.** $\\lambda^2 - 1.1\\lambda + 0.28 = 0 \\Rightarrow \\lambda = \\frac{1.1 \\pm \\sqrt{1.21 - 1.12}}{2} = \\frac{1.1 \\pm 0.3}{2} = 0.7,\\; 0.4$.\n", "\n", "**Step 3.** Both satisfy $|\\lambda| < 1$, so $A^t \\to 0$ and $x_t = A^tx_0 \\to 0$.\n", "\n", "**Answer:** yes; shocks die out, by 30% per period along the slower direction ($\\lambda = 0.7$).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Two almost identical wealth measures\n", "\n", "A poverty study includes both an asset index and household income as regressors. After standardising, they have correlation 0.99, so $X^{\\top}X/n = \\begin{pmatrix}1 & 0.99\\\\0.99 & 1\\end{pmatrix}$. Find its eigenvalues and condition number, and say which combination of the two coefficients the data cannot pin down.\n", "\n", "
Show solution\n", "\n", "**Step 1.** For $\\begin{pmatrix}1 & \\rho\\\\\\rho & 1\\end{pmatrix}$ the eigenvectors are $(1, 1)$ and $(1, -1)$ with eigenvalues $1 + \\rho$ and $1 - \\rho$ (as in Exercise 2).\n", "\n", "**Step 2.** With $\\rho = 0.99$: eigenvalues 1.99 and 0.01.\n", "\n", "**Step 3.** Condition number $= 1.99/0.01 = 199$.\n", "\n", "**Step 4.** The tiny eigenvalue belongs to $(1, -1)$: the data contain almost no variation in \"assets minus income\".\n", "\n", "**Answer:** condition number 199. The *sum* of the two coefficients is well estimated, their *difference* hardly at all (see the collinearity simulator).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Ridge repairs the spectrum\n", "\n", "Add $\\lambda = 0.1$ to the diagonal (ridge regression). What happens to the eigenvalues and condition number?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Adding $\\lambda I$ keeps the eigenvectors and adds $\\lambda$ to each eigenvalue: $(X^{\\top}X/n + 0.1I)q_j = (\\lambda_j + 0.1)q_j$.\n", "\n", "**Step 2.** New eigenvalues: $1.99 + 0.1 = 2.09$ and $0.01 + 0.1 = 0.11$.\n", "\n", "**Step 3.** New condition number: $2.09/0.11 = 19$.\n", "\n", "**Answer:** ridge cuts the condition number from 199 to 19, trading a little bias for a large drop in variance (Day 8).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · How different are two pupils?\n", "\n", "A matching estimator compares a tutored pupil with an untutored one. Their two baseline test scores (maths, reading) differ by $d = (2, 1)$ points, and the scores' covariance matrix is $\\Sigma$ from Exercise 11. Compute the Mahalanobis distance $\\sqrt{d^{\\top}\\Sigma^{-1}d}$ and compare it with the Euclidean distance.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\det\\Sigma = 8$, so $\\Sigma^{-1} = \\frac18\\begin{pmatrix}3 & -2\\\\-2 & 4\\end{pmatrix}$.\n", "\n", "**Step 2.** Quadratic form: $d^{\\top}\\Sigma^{-1}d = \\frac18\\big(3(2)^2 + 2(-2)(2)(1) + 4(1)^2\\big) = \\frac18(12 - 8 + 4) = 1$.\n", "\n", "**Step 3.** Mahalanobis distance $= \\sqrt1 = 1$; Euclidean distance $= \\sqrt{4 + 1} \\approx 2.24$.\n", "\n", "**Answer:** 1 versus 2.24. The difference lies along the high-variance \"general ability\" direction, where such gaps are common, so the two pupils are a good match. Mahalanobis distance is scale-free and accounts for correlation.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "A = np.array([[2, 1], [1, 2.]])\n", "lam, Q = np.linalg.eigh(A)\n", "print(\"eigenvalues:\", lam, \"\\neigenvectors (columns):\\n\", Q.round(3))\n", "print(\"Q diag(lam) Q' == A:\", np.allclose(Q @ np.diag(lam) @ Q.T, A))\n", "\n", "S = np.array([[4, 2], [2, 3.]])\n", "ev = np.linalg.eigvalsh(S)[::-1]\n", "print(\"PC variances:\", ev.round(3), \" first PC share:\", round(ev[0] / ev.sum(), 3))\n", "\n", "D = np.array([[0.5, 0.2], [0.1, 0.6]])\n", "print(\"dynamic eigenvalues:\", np.sort(np.linalg.eigvals(D)).round(3))\n", "\n", "C = np.array([[1, 0.99], [0.99, 1]])\n", "print(\"condition number:\", round(np.linalg.cond(C)), \" with ridge 0.1:\", round(np.linalg.cond(C + 0.1 * np.eye(2))))\n", "d = np.array([2, 1.])\n", "print(\"Mahalanobis:\", np.sqrt(d @ np.linalg.solve(S, d)), \" Euclidean:\", round(np.linalg.norm(d), 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Eigenvectors are the directions a matrix only stretches; eigenvalues are the stretch factors.\n", "- Symmetric matrices diagonalise in an orthonormal basis, so definiteness is read off the eigenvalues.\n", "- Eigenvalues tell you the main directions of variance, how collinear your regressors are, and whether a dynamic system is stable." ] } ] }