{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 22: Positive-Definite Matrices & Variance–Covariance Matrices\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Random vectors, their covariance matrices, and the rules for the variance of any linear combination.\n", "\n", "**Learning outcomes**\n", "\n", "- Compute means and covariance matrices of random vectors and the variance of any linear combination a′X.\n", "- Prove Var(AX + b) = AΣA′ and that every covariance matrix is positive semi-definite.\n", "- Use the Cholesky factor to simulate correlated variables.\n", "- Compute standard errors of sums and differences of correlated coefficient estimates." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Random vectors\n", "\n", "For a random vector $X = (X_1, \\dots, X_k)^{\\top}$, the **mean vector** is $\\mu = E[X]$ and the **variance–covariance matrix** is\n", "\n", "$$\n", "\\Sigma = \\operatorname{Var}(X) = E\\big[(X - \\mu)(X - \\mu)^{\\top}\\big], \\qquad \\Sigma_{jj} = \\operatorname{Var}(X_j), \\quad \\Sigma_{j\\ell} = \\operatorname{Cov}(X_j, X_\\ell).\n", "$$\n", "\n", "$\\Sigma$ is symmetric. Its correlation matrix is $D^{-1/2}\\Sigma D^{-1/2}$ with $D = \\operatorname{diag}(\\Sigma)$.\n", "\n", "### The two rules you will use every week\n", "\n", "$$\n", "\\operatorname{Var}(a^{\\top}X) = a^{\\top}\\Sigma a, \\qquad \\operatorname{Var}(AX + b) = A\\Sigma A^{\\top}.\n", "$$\n", "\n", "They give the variance of a portfolio, a contrast between treatment arms, a cumulative event-study effect, or a predicted value $x_0^{\\top}\\hat\\beta$: just plug in $\\operatorname{Var}(\\hat\\beta)$.\n", "\n", "### Positive (semi-)definiteness\n", "\n", "Because $a^{\\top}\\Sigma a$ is a variance, it can never be negative: **every covariance matrix is PSD**. It is PD unless some combination of the variables is constant (e.g. shares that sum to one). A symmetric matrix that is not PSD (e.g. a \"correlation matrix\" assembled pairwise from different samples) cannot be the covariance matrix of anything.\n", "\n", "### Cholesky factor and simulation\n", "\n", "A PD matrix factors as $\\Sigma = LL^{\\top}$ with $L$ lower-triangular (the **Cholesky** factor). If $Z \\sim N(0, I)$, then\n", "\n", "$$\n", "X = \\mu + LZ \\sim N(\\mu, \\Sigma).\n", "$$\n", "\n", "This is how simulations generate correlated covariates, errors or potential outcomes.\n", "\n", "### Sample covariance\n", "\n", "With the centred data matrix $\\tilde X$ ($n \\times k$), $S = \\frac{1}{n - 1}\\tilde X^{\\top}\\tilde X$, which is PSD by Day 17.\n", "\n", "## Part 1 · Warm-up\n", "\n", "Let $\\Sigma = \\begin{pmatrix}4 & 2\\\\2 & 9\\end{pmatrix}$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Variance of a sum\n", "\n", "Find $\\operatorname{Var}(X_1 + X_2)$.\n", "\n", "
Show solution\n", "\n", "$a = (1, 1)$: $a^{\\top}\\Sigma a = 4 + 9 + 2(2) = 17$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Variance of a difference\n", "\n", "Find $\\operatorname{Var}(X_1 - X_2)$.\n", "\n", "
Show solution\n", "\n", "$a = (1, -1)$: $4 + 9 - 2(2) = 9$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Correlation\n", "\n", "Find the correlation between $X_1$ and $X_2$.\n", "\n", "
Show solution\n", "\n", "$2/(\\sqrt4\\sqrt9) = 2/6 \\approx 0.333$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · A valid covariance matrix?\n", "\n", "Can $\\begin{pmatrix}1 & 2\\\\2 & 1\\end{pmatrix}$ be a covariance matrix?\n", "\n", "
Show solution\n", "\n", "**Step 1.** No.\n", "\n", "**Step 2.** Its determinant is $-3 < 0$, so it has a negative eigenvalue; equivalently the implied correlation is 2.\n", "\n", "**Step 3.** For $a = (1, -1)$, $a^{\\top}\\Sigma a = 1 + 1 - 4 = -2$: a negative variance.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Cholesky by hand\n", "\n", "Find the Cholesky factor of $\\Sigma$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Write $L = \\begin{pmatrix}\\ell_{11} & 0\\\\\\ell_{21} & \\ell_{22}\\end{pmatrix}$, so $LL^{\\top} = \\begin{pmatrix}\\ell_{11}^2 & \\ell_{11}\\ell_{21}\\\\\\ell_{11}\\ell_{21} & \\ell_{21}^2 + \\ell_{22}^2\\end{pmatrix}$.\n", "\n", "**Step 2.** Match to $\\Sigma$: $\\ell_{11}^2 = 4 \\Rightarrow \\ell_{11} = 2$.\n", "\n", "**Step 3.** $\\ell_{11}\\ell_{21} = 2 \\Rightarrow \\ell_{21} = 1$.\n", "\n", "**Step 4.** $\\ell_{21}^2 + \\ell_{22}^2 = 9 \\Rightarrow \\ell_{22} = \\sqrt8 \\approx 2.828$.\n", "\n", "**Answer:** $L = \\begin{pmatrix}2 & 0\\\\1 & 2.828\\end{pmatrix}$.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Variance of a linear combination\n", "\n", "Prove $\\operatorname{Var}(a^{\\top}X) = a^{\\top}\\Sigma a$.\n", "\n", "
Show solution\n", "\n", "$a^{\\top}X - E[a^{\\top}X] = a^{\\top}(X - \\mu)$, a scalar equal to its own transpose. So\n", "\n", "$$\n", "\\operatorname{Var}(a^{\\top}X) = E\\big[a^{\\top}(X - \\mu)(X - \\mu)^{\\top}a\\big] = a^{\\top}E\\big[(X - \\mu)(X - \\mu)^{\\top}\\big]a = a^{\\top}\\Sigma a.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Linear transformations\n", "\n", "Prove $\\operatorname{Var}(AX + b) = A\\Sigma A^{\\top}$.\n", "\n", "
Show solution\n", "\n", "$AX + b - E[AX + b] = A(X - \\mu)$, so $\\operatorname{Var} = E[A(X - \\mu)(X - \\mu)^{\\top}A^{\\top}] = A\\Sigma A^{\\top}$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Covariance matrices are PSD\n", "\n", "Prove that every covariance matrix is positive semi-definite.\n", "\n", "
Show solution\n", "\n", "For any $a$, $a^{\\top}\\Sigma a = \\operatorname{Var}(a^{\\top}X) \\ge 0$ (Exercise 6).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Simulating with Cholesky\n", "\n", "Prove that if $Z \\sim N(0, I)$ then $X = \\mu + LZ$ has mean $\\mu$ and covariance $LL^{\\top} = \\Sigma$.\n", "\n", "
Show solution\n", "\n", "$E[X] = \\mu + LE[Z] = \\mu$, and by Exercise 7, $\\operatorname{Var}(X) = L\\,I\\,L^{\\top} = \\Sigma$. (A linear function of normals is normal, so $X \\sim N(\\mu, \\Sigma)$.)\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · What PD implies about entries\n", "\n", "Show that a PD matrix has positive diagonal entries and $|\\sigma_{12}| < \\sqrt{\\sigma_{11}\\sigma_{22}}$.\n", "\n", "
Show solution\n", "\n", "Take $a = e_j$ (the $j$th unit vector): $\\sigma_{jj} = e_j^{\\top}\\Sigma e_j > 0$. The 2×2 leading block is also PD, so its determinant (product of its eigenvalues) is positive: $\\sigma_{11}\\sigma_{22} - \\sigma_{12}^2 > 0$.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Portfolio risk\n", "\n", "Two assets have return sds 10% and 20% and correlation 0.3. With weights $w = (0.6, 0.4)$, find the portfolio sd.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Covariance: $0.3 \\times 10 \\times 20 = 60$, so $\\Sigma = \\begin{pmatrix}100 & 60\\\\60 & 400\\end{pmatrix}$.\n", "\n", "**Step 2.** $w^{\\top}\\Sigma w = 0.6^2(100) + 0.4^2(400) + 2(0.6)(0.4)(60) = 36 + 64 + 28.8 = 128.8$.\n", "\n", "**Step 3.** sd $= \\sqrt{128.8} \\approx 11.35$.\n", "\n", "**Answer:** portfolio sd ≈ 11.35%.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Minimum-variance weights\n", "\n", "For the same assets, find the weight $w_1$ that minimises portfolio variance (with $w_2 = 1 - w_1$).\n", "\n", "
Show solution\n", "\n", "**Step 1.** Variance as a function of $w_1$: $V(w_1) = w_1^2\\sigma_1^2 + (1 - w_1)^2\\sigma_2^2 + 2w_1(1 - w_1)\\sigma_{12}$.\n", "\n", "**Step 2.** Set $V'(w_1) = 2w_1\\sigma_1^2 - 2(1 - w_1)\\sigma_2^2 + 2(1 - 2w_1)\\sigma_{12} = 0$ and solve:\n", "\n", "$$\n", "w_1 = \\frac{\\sigma_2^2 - \\sigma_{12}}{\\sigma_1^2 + \\sigma_2^2 - 2\\sigma_{12}}.\n", "$$\n", "\n", "**Step 3.** Plug in: $w_1 = (400 - 60)/(100 + 400 - 120) = 340/380 \\approx 0.895$.\n", "\n", "**Answer:** about 89.5% in the less risky asset (Day 26 derives the general version with a Lagrange multiplier).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Are two effects different?\n", "\n", "A regression gives $\\hat\\beta_1 = 0.5$ and $\\hat\\beta_2 = 0.2$ with $\\operatorname{Var}(\\hat\\beta_1) = 0.04$, $\\operatorname{Var}(\\hat\\beta_2) = 0.09$, $\\operatorname{Cov} = 0.02$. Test $\\beta_1 = \\beta_2$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Contrast $a = (1, -1)$: $\\operatorname{Var}(\\hat\\beta_1 - \\hat\\beta_2) = 0.04 + 0.09 - 2(0.02) = 0.09$.\n", "\n", "**Step 2.** SE $= 0.3$; estimate $= 0.5 - 0.2 = 0.3$.\n", "\n", "**Step 3.** $z = 0.3/0.3 = 1$, $p \\approx 0.32$.\n", "\n", "**Answer:** not significantly different. Ignoring the covariance would give SE $\\sqrt{0.13} \\approx 0.36$ instead.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · A cumulative effect\n", "\n", "An event study estimates three post-period effects, each with variance 0.01 and pairwise covariance 0.005. Find the SE of their sum.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The sum is $\\mathbf 1^{\\top}\\hat\\beta$, so its variance is $\\mathbf 1^{\\top}V\\mathbf 1$ = the sum of *all* entries of $V$.\n", "\n", "**Step 2.** Diagonal: $3 \\times 0.01 = 0.03$. Off-diagonal: 6 entries × 0.005 = 0.03.\n", "\n", "**Step 3.** Total 0.06, SE $= \\sqrt{0.06} \\approx 0.245$.\n", "\n", "**Answer:** 0.245; assuming independence would give 0.173, far too small.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · An impossible set of survey correlations\n", "\n", "A report on a household survey lists pairwise correlations computed from different sub-samples: income–education 0.9, income–health 0.9, and education–health −0.9. Could these be the correlations of three real variables?\n", "\n", "
Show solution\n", "\n", "**Step 1.** A valid correlation matrix must be PSD, so its determinant (the product of the eigenvalues) cannot be negative.\n", "\n", "**Step 2.** Cofactor expansion of $R = \\begin{pmatrix}1 & 0.9 & 0.9\\\\0.9 & 1 & -0.9\\\\0.9 & -0.9 & 1\\end{pmatrix}$:\n", "\n", "$$\n", "\\det R = 1(1 - 0.81) - 0.9(0.9 + 0.81) + 0.9(-0.81 - 0.9) = 0.19 - 1.539 - 1.539 \\approx -2.89.\n", "$$\n", "\n", "**Step 3.** Negative, so $R$ has a negative eigenvalue (−0.8, from the code).\n", "\n", "**Answer:** impossible. If income is strongly correlated with both education and health, those two cannot be strongly negatively correlated. Pairwise estimates from different samples can produce such incoherent matrices.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "S = np.array([[4, 2], [2, 9.]])\n", "for a in ([1, 1], [1, -1]):\n", " a = np.array(a); print(\"Var(a'X) for a =\", a, \":\", a @ S @ a)\n", "L = np.linalg.cholesky(S); print(\"Cholesky L =\\n\", L.round(3))\n", "\n", "rng = np.random.default_rng(22)\n", "Z = rng.normal(size=(2, 200_000)); X = (L @ Z).T\n", "print(\"simulated covariance:\\n\", np.cov(X.T).round(2))\n", "\n", "w = np.array([0.6, 0.4]); Sig = np.array([[100, 60], [60, 400.]])\n", "print(\"portfolio sd:\", round(np.sqrt(w @ Sig @ w), 2))\n", "V = np.array([[0.01, 0.005, 0.005], [0.005, 0.01, 0.005], [0.005, 0.005, 0.01]])\n", "print(\"SE of cumulative effect:\", round(np.sqrt(np.ones(3) @ V @ np.ones(3)), 3))\n", "R = np.array([[1, 0.9, 0.9], [0.9, 1, -0.9], [0.9, -0.9, 1]])\n", "print(\"eigenvalues of the 'correlation matrix':\", np.linalg.eigvalsh(R).round(3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- $\\operatorname{Var}(a^{\\top}X) = a^{\\top}\\Sigma a$ and $\\operatorname{Var}(AX + b) = A\\Sigma A^{\\top}$ give the variance of any linear combination.\n", "- Covariance matrices are PSD; a matrix that is not PSD cannot describe real data.\n", "- Cholesky factors turn independent normals into correlated ones, the basis of most simulation studies." ] } ] }