{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 5: Joint Distributions, Covariance & Correlation\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "How two variables move together, and why that is the raw material of regression and omitted-variable bias.\n", "\n", "**Learning outcomes**\n", "\n", "- Read marginal and conditional distributions off a joint table.\n", "- Compute and interpret covariance and correlation, and prove their key properties.\n", "- Show that a regression slope on a binary regressor equals a difference in means.\n", "- Derive the omitted-variable-bias formula in its simplest form." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Joint, marginal and conditional\n", "\n", "For two discrete variables, the **joint pmf** is $p(x, y) = P(X = x, Y = y)$. Summing over one variable gives the **marginal**, $p_X(x) = \\sum_y p(x, y)$, and dividing gives the **conditional**, $p(y \\mid x) = p(x, y)/p_X(x)$. $X$ and $Y$ are **independent** when $p(x, y) = p_X(x)p_Y(y)$ for all $x, y$. Continuous variables work the same way with densities and integrals.\n", "\n", "### Covariance and correlation\n", "\n", "$$\n", "\\operatorname{Cov}(X, Y) = E\\big[(X - \\mu_X)(Y - \\mu_Y)\\big] = E[XY] - E[X]E[Y], \\qquad\n", "\\rho_{XY} = \\frac{\\operatorname{Cov}(X, Y)}{\\sigma_X \\sigma_Y} \\in [-1, 1].\n", "$$\n", "\n", "Covariance has units (dollars × years); correlation is unit-free. Useful rules:\n", "\n", "- $\\operatorname{Cov}(X, X) = \\operatorname{Var}(X)$\n", "- $\\operatorname{Cov}(aX + b, cY + d) = ac\\,\\operatorname{Cov}(X, Y)$\n", "- $\\operatorname{Var}(X \\pm Y) = \\operatorname{Var}(X) + \\operatorname{Var}(Y) \\pm 2\\operatorname{Cov}(X, Y)$\n", "- Independence ⇒ zero covariance, but **not** the other way round.\n", "\n", "### Covariance is the regression slope's numerator\n", "\n", "The best linear predictor of $Y$ given $X$ has slope\n", "\n", "$$\n", "\\beta = \\frac{\\operatorname{Cov}(X, Y)}{\\operatorname{Var}(X)}.\n", "$$\n", "\n", "If $Y = \\beta X + \\gamma U + \\varepsilon$ and you leave $U$ out, the slope you estimate is $\\beta + \\gamma\\,\\operatorname{Cov}(X, U)/\\operatorname{Var}(X)$: the famous **omitted-variable bias**.\n", "\n", "### The bivariate normal\n", "\n", "If $(X, Y)$ is bivariate normal with correlation $\\rho$, the conditional mean is linear and the conditional variance is constant:\n", "\n", "$$\n", "E[Y \\mid X = x] = \\mu_Y + \\rho\\frac{\\sigma_Y}{\\sigma_X}(x - \\mu_X), \\qquad \\operatorname{Var}(Y \\mid X) = \\sigma_Y^2(1 - \\rho^2).\n", "$$\n", "\n", "Explore it in 3D in the Simulation Lab.\n", "\n", "## Part 1 · Warm-up\n", "\n", "The joint distribution of job training $X$ and employment $Y$ is:\n", "\n", "| | $Y = 0$ | $Y = 1$ |\n", "|---|---|---|\n", "| $X = 0$ | 0.30 | 0.30 |\n", "| $X = 1$ | 0.10 | 0.30 |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Marginals\n", "\n", "Find $P(X = 1)$ and $P(Y = 1)$.\n", "\n", "
Show solution\n", "\n", "$P(X = 1) = 0.10 + 0.30 = 0.40$ and $P(Y = 1) = 0.30 + 0.30 = 0.60$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · The product moment\n", "\n", "Compute $E[XY]$.\n", "\n", "
Show solution\n", "\n", "$XY = 1$ only when both are 1, so $E[XY] = P(X = 1, Y = 1) = 0.30$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Covariance\n", "\n", "Compute $\\operatorname{Cov}(X, Y)$.\n", "\n", "
Show solution\n", "\n", "$\\operatorname{Cov}(X, Y) = 0.30 - 0.40 \\times 0.60 = 0.06$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Correlation\n", "\n", "Compute $\\rho_{XY}$.\n", "\n", "
Show solution\n", "\n", "$\\operatorname{Var}(X) = 0.4 \\times 0.6 = 0.24$ and $\\operatorname{Var}(Y) = 0.6 \\times 0.4 = 0.24$, so $\\rho = 0.06/\\sqrt{0.24 \\times 0.24} = 0.25$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Conditionals and independence\n", "\n", "Find $P(Y = 1 \\mid X = 1)$ and $P(Y = 1 \\mid X = 0)$. Are $X$ and $Y$ independent?\n", "\n", "
Show solution\n", "\n", "$P(Y = 1 \\mid X = 1) = 0.30/0.40 = 0.75$ and $P(Y = 1 \\mid X = 0) = 0.30/0.60 = 0.50$. They differ, so $X$ and $Y$ are not independent (also, $0.30 \\ne 0.40 \\times 0.60$).\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Covariance shortcut\n", "\n", "Prove $\\operatorname{Cov}(X, Y) = E[XY] - E[X]E[Y]$.\n", "\n", "
Show solution\n", "\n", "$$\n", "E[(X - \\mu_X)(Y - \\mu_Y)] = E[XY] - \\mu_Y E[X] - \\mu_X E[Y] + \\mu_X\\mu_Y = E[XY] - \\mu_X\\mu_Y.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Independence implies zero covariance\n", "\n", "Prove that if $X$ and $Y$ are independent (discrete), $\\operatorname{Cov}(X, Y) = 0$.\n", "\n", "
Show solution\n", "\n", "$$\n", "E[XY] = \\sum_x\\sum_y xy\\,p_X(x)p_Y(y) = \\Big(\\sum_x x\\,p_X(x)\\Big)\\Big(\\sum_y y\\,p_Y(y)\\Big) = E[X]E[Y],\n", "$$\n", "\n", "so the covariance is zero.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Variance of a sum\n", "\n", "Prove $\\operatorname{Var}(X + Y) = \\operatorname{Var}(X) + \\operatorname{Var}(Y) + 2\\operatorname{Cov}(X, Y)$.\n", "\n", "
Show solution\n", "\n", "Let $\\tilde X = X - \\mu_X$ and $\\tilde Y = Y - \\mu_Y$. Then\n", "\n", "$$\n", "\\operatorname{Var}(X + Y) = E[(\\tilde X + \\tilde Y)^2] = E[\\tilde X^2] + E[\\tilde Y^2] + 2E[\\tilde X\\tilde Y].\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Correlation is bounded\n", "\n", "Prove $|\\rho_{XY}| \\le 1$. *Hint: consider $\\operatorname{Var}(Y - tX) \\ge 0$ for every $t$.*\n", "\n", "
Show solution\n", "\n", "**Step 1.** $h(t) = \\operatorname{Var}(Y - tX) = \\sigma_Y^2 - 2t\\,\\sigma_{XY} + t^2\\sigma_X^2 \\ge 0$ for all $t$.\n", "\n", "**Step 2.** A quadratic in $t$ that never goes negative has a non-positive discriminant:\n", "\n", "$$\n", "4\\sigma_{XY}^2 - 4\\sigma_X^2\\sigma_Y^2 \\le 0 \\;\\Rightarrow\\; \\sigma_{XY}^2 \\le \\sigma_X^2\\sigma_Y^2 \\;\\Rightarrow\\; |\\rho| \\le 1.\n", "$$\n", "\n", "**Step 3.** Equality holds exactly when $Y - tX$ is constant for some $t$, i.e. when $Y$ is a linear function of $X$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · The slope on a binary regressor\n", "\n", "Let $X \\in \\{0, 1\\}$ with $P(X = 1) = p$. Prove\n", "\n", "$$\n", "\\frac{\\operatorname{Cov}(X, Y)}{\\operatorname{Var}(X)} = E[Y \\mid X = 1] - E[Y \\mid X = 0].\n", "$$\n", "\n", "
Show solution\n", "\n", "**Step 1.** Let $m_1 = E[Y \\mid X = 1]$ and $m_0 = E[Y \\mid X = 0]$.\n", "\n", "**Step 2.** Then $E[XY] = p\\,m_1$, $E[X] = p$ and $E[Y] = p\\,m_1 + (1 - p)m_0$.\n", "\n", "**Step 3.** So\n", "\n", "$$\n", "\\operatorname{Cov}(X, Y) = p\\,m_1 - p\\big(p\\,m_1 + (1 - p)m_0\\big) = p(1 - p)(m_1 - m_0).\n", "$$\n", "\n", "**Step 4.** Dividing by $\\operatorname{Var}(X) = p(1 - p)$ gives $m_1 - m_0$.\n", "\n", "**Step 5.** For the table above: $0.06/0.24 = 0.25 = 0.75 - 0.50$.\n", "\n", "**Step 6.** Regressing an outcome on a treatment dummy *is* a difference in means.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · A U-shaped health risk\n", "\n", "Daily temperature deviation from the seasonal norm, $X$, is $-1$ (cold snap), 0 or $+1$ (heat wave), each with probability 1/3. Excess hospital admissions are $Y = X^2$ (both extremes raise admissions). Show that $\\operatorname{Cov}(X, Y) = 0$ although $Y$ is completely determined by $X$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $E[X] = (-1 + 0 + 1)/3 = 0$.\n", "\n", "**Step 2.** $XY = X^3$, so $E[XY] = ((-1)^3 + 0 + 1^3)/3 = 0$.\n", "\n", "**Step 3.** $\\operatorname{Cov}(X, Y) = E[XY] - E[X]E[Y] = 0 - 0 \\cdot E[Y] = 0$.\n", "\n", "**Answer:** the covariance (and correlation) is zero, yet temperature determines admissions exactly. Covariance only measures *linear* association; a linear regression of admissions on temperature would wrongly suggest no relationship.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Diversifying programmes\n", "\n", "A foundation splits its budget equally between programmes A and B with impact standard deviations 10 and 20 and correlation $-0.5$. Find the standard deviation of total impact $0.5A + 0.5B$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Covariance of the two programmes: $\\operatorname{Cov}(A, B) = \\rho\\sigma_A\\sigma_B = -0.5 \\times 10 \\times 20 = -100$.\n", "\n", "**Step 2.** Variance of the half-and-half portfolio:\n", "\n", "$$\n", "\\operatorname{Var}(0.5A + 0.5B) = 0.25(100) + 0.25(400) + 2(0.5)(0.5)(-100) = 25 + 100 - 50 = 75.\n", "$$\n", "\n", "**Step 3.** sd $= \\sqrt{75} \\approx 8.66$.\n", "\n", "**Answer:** 8.66, below the sd of either programme alone (10 and 20): negatively correlated programmes hedge each other.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Omitted ability in a wage regression\n", "\n", "Log wages depend on schooling $X$ (years, standardised so $\\operatorname{Var}(X) = 1$) and ability $U$: $Y = 2X + 3U + \\varepsilon$, with $\\varepsilon$ uncorrelated with $X$ and $U$, and $\\operatorname{Cov}(X, U) = 0.5$ (abler people stay in school longer). What slope do you get from regressing $Y$ on $X$ alone?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The short-regression slope is $\\operatorname{Cov}(X, Y)/\\operatorname{Var}(X)$.\n", "\n", "**Step 2.** Expand the covariance using the model:\n", "\n", "$$\n", "\\operatorname{Cov}(X, Y) = 2\\operatorname{Var}(X) + 3\\operatorname{Cov}(X, U) + \\operatorname{Cov}(X, \\varepsilon) = 2(1) + 3(0.5) + 0 = 3.5.\n", "$$\n", "\n", "**Step 3.** Divide by $\\operatorname{Var}(X) = 1$: slope $= 3.5$.\n", "\n", "**Step 4.** Bias $= 3.5 - 2 = 1.5$ = (effect of ability on wages, 3) × (slope of ability on schooling, 0.5).\n", "\n", "**Answer:** 3.5. Leaving out ability overstates the return to schooling by 75%.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · When is a training coefficient causal?\n", "\n", "Earnings follow $Y = \\beta_0 + \\beta_1X + \\varepsilon$, where $X$ is hours of job training and $\\varepsilon$ collects every other cause of earnings (motivation, local labour demand, …). Show that the population regression slope $\\operatorname{Cov}(X, Y)/\\operatorname{Var}(X)$ equals $\\beta_1$ if and only if $\\operatorname{Cov}(X, \\varepsilon) = 0$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Take the covariance of both sides with $X$ (constants have zero covariance):\n", "\n", "$$\n", "\\operatorname{Cov}(X, Y) = \\beta_1\\operatorname{Var}(X) + \\operatorname{Cov}(X, \\varepsilon).\n", "$$\n", "\n", "**Step 2.** Divide by $\\operatorname{Var}(X)$:\n", "\n", "$$\n", "\\frac{\\operatorname{Cov}(X, Y)}{\\operatorname{Var}(X)} = \\beta_1 + \\frac{\\operatorname{Cov}(X, \\varepsilon)}{\\operatorname{Var}(X)}.\n", "$$\n", "\n", "**Step 3.** The second term vanishes exactly when $\\operatorname{Cov}(X, \\varepsilon) = 0$.\n", "\n", "**Answer:** the slope is causal only if training hours are uncorrelated with all other causes of earnings. If motivated workers take more training, the slope overstates $\\beta_1$. This **exogeneity** condition is what every identification strategy in Phase 5 tries to secure.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Prediction with a bivariate normal\n", "\n", "Years of education $X$ have mean 12 and sd 2; log wages $Y$ have mean 3 and sd 0.5; $\\rho = 0.4$, jointly normal. Find $E[Y \\mid X = 16]$ and $\\operatorname{sd}(Y \\mid X)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Conditional mean of a bivariate normal:\n", "\n", "$$\n", "E[Y \\mid X = 16] = \\mu_Y + \\rho\\frac{\\sigma_Y}{\\sigma_X}(16 - \\mu_X) = 3 + 0.4 \\times \\frac{0.5}{2} \\times 4 = 3 + 0.4 = 3.4.\n", "$$\n", "\n", "**Step 2.** Conditional variance: $\\operatorname{Var}(Y \\mid X) = \\sigma_Y^2(1 - \\rho^2) = 0.25 \\times 0.84 = 0.21$.\n", "\n", "**Step 3.** Conditional sd: $\\sqrt{0.21} \\approx 0.458$, versus the unconditional 0.5.\n", "\n", "**Answer:** predicted log wage 3.4 for 16 years of schooling, with sd 0.458. A correlation of 0.4 removes only 16% of the variance; and this slope is a *predictive* relationship, not a causal return to schooling.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "joint = np.array([[0.30, 0.30], # rows: X = 0, 1; columns: Y = 0, 1\n", " [0.10, 0.30]])\n", "x = np.array([0, 1]); y = np.array([0, 1])\n", "px, py = joint.sum(1), joint.sum(0)\n", "exy = sum(joint[i, j] * x[i] * y[j] for i in range(2) for j in range(2))\n", "cov = exy - (px @ x) * (py @ y)\n", "vx, vy = px @ x**2 - (px @ x)**2, py @ y**2 - (py @ y)**2\n", "print(f\"Cov = {cov:.2f}, corr = {cov / np.sqrt(vx * vy):.2f}, slope = {cov / vx:.2f}\")\n", "\n", "# Omitted-variable bias by simulation (Exercise 13)\n", "rng = np.random.default_rng(7)\n", "n = 500_000\n", "X = rng.normal(0, 1, n)\n", "U = 0.5 * X + rng.normal(0, 1, n) # Cov(X, U) = 0.5\n", "Y = 2 * X + 3 * U + rng.normal(0, 1, n)\n", "print(\"short-regression slope:\", round(np.cov(X, Y)[0, 1] / X.var(ddof=1), 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Covariance measures linear co-movement; correlation rescales it to $[-1, 1]$.\n", "- A regression slope is a covariance divided by a variance; on a dummy it is a difference in means.\n", "- Leaving out a variable correlated with $X$ biases the slope by (its effect) × (its slope on $X$)." ] } ] }