{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 3: Random Variables, Expectation & Variance\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Turning outcomes into numbers, and the two summaries every estimator is built from.\n", "\n", "**Learning outcomes**\n", "\n", "- Compute expectations and variances for discrete and continuous random variables.\n", "- Prove linearity of expectation and the variance rules for linear transformations.\n", "- Define the average treatment effect as an expectation of potential outcomes.\n", "- Use Jensen's inequality to reason about non-linear functions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Random variables\n", "\n", "A **random variable** $X$ is a function from the sample space to the real numbers, $X: \\Omega \\to \\mathbb{R}$. It turns outcomes (\"the firm adopted clean tech\") into numbers (emissions in tonnes). Its distribution is described by a **probability mass function** $p(x) = P(X = x)$ (discrete) or a **density** $f(x)$ (continuous), with $P(a \\le X \\le b) = \\int_a^b f(x)\\,dx$.\n", "\n", "### Expectation\n", "\n", "The **expected value** is the probability-weighted average:\n", "\n", "$$\n", "E[X] = \\sum_x x\\,p(x) \\qquad\\text{or}\\qquad E[X] = \\int_{-\\infty}^{\\infty} x f(x)\\,dx.\n", "$$\n", "\n", "For any function $g$, $E[g(X)] = \\sum_x g(x)p(x)$ (the \"law of the unconscious statistician\").\n", "\n", "**Linearity of expectation** holds for *any* random variables, dependent or not:\n", "\n", "$$\n", "E[aX + bY + c] = aE[X] + bE[Y] + c.\n", "$$\n", "\n", "### Variance\n", "\n", "$$\n", "\\operatorname{Var}(X) = E\\big[(X - E[X])^2\\big] = E[X^2] - (E[X])^2, \\qquad \\operatorname{Var}(aX + b) = a^2 \\operatorname{Var}(X).\n", "$$\n", "\n", "The standard deviation $\\operatorname{sd}(X) = \\sqrt{\\operatorname{Var}(X)}$ is in the units of $X$. Shifting a variable moves its centre but not its spread; scaling by $a$ scales the spread by $|a|$.\n", "\n", "### Jensen's inequality\n", "\n", "If $g$ is convex, $E[g(X)] \\ge g(E[X])$; if concave, the inequality flips. In general $E[g(X)] \\ne g(E[X])$, which is why the average of logs is not the log of the average (Day 12 builds on this).\n", "\n", "### Potential outcomes as random variables\n", "\n", "Each unit has two **potential outcomes**: $Y(1)$ if treated and $Y(0)$ if not. The **average treatment effect** is an expectation:\n", "\n", "$$\n", "\\tau_{ATE} = E[Y(1) - Y(0)] = E[Y(1)] - E[Y(0)],\n", "$$\n", "\n", "where the second equality is linearity. We will never observe $Y(1) - Y(0)$ for anyone, but linearity tells us we only need the two *means* separately.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Expected fine\n", "\n", "A random inspection yields a fine of \\$0 with probability 0.6, \\$500 with probability 0.3 and \\$2,000 with probability 0.1. Find $E[X]$.\n", "\n", "
Show solution\n", "\n", "$$\n", "E[X] = 0(0.6) + 500(0.3) + 2000(0.1) = 0 + 150 + 200 = 350.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · A second moment\n", "\n", "For the same fine, compute $E[X^2]$.\n", "\n", "
Show solution\n", "\n", "$$\n", "E[X^2] = 0^2(0.6) + 500^2(0.3) + 2000^2(0.1) = 75{,}000 + 400{,}000 = 475{,}000.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Variance by the shortcut\n", "\n", "Compute $\\operatorname{Var}(X)$ and $\\operatorname{sd}(X)$.\n", "\n", "
Show solution\n", "\n", "$$\n", "\\operatorname{Var}(X) = 475{,}000 - 350^2 = 475{,}000 - 122{,}500 = 352{,}500,\n", "$$\n", "\n", "so $\\operatorname{sd}(X) = \\sqrt{352{,}500} \\approx 593.7$ dollars.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Linearity\n", "\n", "Tax revenue is $Y = 15X - 100$, where $X$ is emissions with $E[X] = 1{,}200$ tonnes. Find $E[Y]$.\n", "\n", "
Show solution\n", "\n", "$E[Y] = 15(1200) - 100 = 17{,}900$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Variance of a linear transformation\n", "\n", "If $\\operatorname{Var}(X) = 400$, what are $\\operatorname{Var}(Y)$ and $\\operatorname{sd}(Y)$?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Use $\\operatorname{Var}(aX + b) = a^2\\operatorname{Var}(X)$ with $a = 15$, $b = -100$:\n", "\n", "$$\n", "\\operatorname{Var}(Y) = 15^2 \\times 400 = 225 \\times 400 = 90{,}000.\n", "$$\n", "\n", "**Step 2.** Standard deviation: $\\operatorname{sd}(Y) = |15| \\times \\sqrt{400} = 15 \\times 20 = 300$.\n", "\n", "**Answer:** variance 90,000 and sd 300. The constant $-100$ shifts revenue but does not change its spread.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Linearity with constants\n", "\n", "Prove $E[aX + b] = aE[X] + b$ for a discrete $X$.\n", "\n", "
Show solution\n", "\n", "$$\n", "E[aX + b] = \\sum_x (ax + b)p(x) = a\\sum_x x\\,p(x) + b\\sum_x p(x) = aE[X] + b,\n", "$$\n", "\n", "using $\\sum_x p(x) = 1$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · The variance shortcut\n", "\n", "Prove $\\operatorname{Var}(X) = E[X^2] - (E[X])^2$.\n", "\n", "
Show solution\n", "\n", "Let $\\mu = E[X]$. Expand the square and use linearity:\n", "\n", "$$\n", "E[(X - \\mu)^2] = E[X^2] - 2\\mu E[X] + \\mu^2 = E[X^2] - 2\\mu^2 + \\mu^2 = E[X^2] - \\mu^2.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Variance scaling\n", "\n", "Prove $\\operatorname{Var}(aX + b) = a^2\\operatorname{Var}(X)$.\n", "\n", "
Show solution\n", "\n", "$E[aX + b] = a\\mu + b$, so\n", "\n", "$$\n", "\\operatorname{Var}(aX + b) = E\\big[(aX + b - a\\mu - b)^2\\big] = E\\big[a^2 (X - \\mu)^2\\big] = a^2\\operatorname{Var}(X).\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Indicator variables\n", "\n", "Let $\\mathbf{1}_A$ equal 1 if $A$ happens and 0 otherwise. Show that $E[\\mathbf{1}_A] = P(A)$ and $\\operatorname{Var}(\\mathbf{1}_A) = P(A)(1 - P(A))$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The indicator takes the value 1 with probability $P(A)$ and 0 otherwise:\n", "\n", "$$\n", "E[\\mathbf{1}_A] = 1 \\cdot P(A) + 0 \\cdot P(A^c) = P(A).\n", "$$\n", "\n", "**Step 2.** Since $1^2 = 1$ and $0^2 = 0$, $\\mathbf{1}_A^2 = \\mathbf{1}_A$, so $E[\\mathbf{1}_A^2] = P(A)$.\n", "\n", "**Step 3.** Variance shortcut:\n", "\n", "$$\n", "\\operatorname{Var}(\\mathbf{1}_A) = E[\\mathbf{1}_A^2] - (E[\\mathbf{1}_A])^2 = P(A) - P(A)^2 = P(A)(1 - P(A)).\n", "$$\n", "\n", "This is why a share (e.g. \"share treated\") is simply the mean of a 0/1 variable.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · The mean is the best constant prediction\n", "\n", "Show that $E[(X - c)^2] = \\operatorname{Var}(X) + (E[X] - c)^2$, and conclude that $c = E[X]$ minimises the mean squared error.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Write $X - c = (X - \\mu) + (\\mu - c)$ and expand:\n", "\n", "$$\n", "E[(X - c)^2] = E[(X - \\mu)^2] + 2(\\mu - c)\\underbrace{E[X - \\mu]}_{=0} + (\\mu - c)^2 = \\operatorname{Var}(X) + (\\mu - c)^2.\n", "$$\n", "\n", "**Step 2.** The second term is non-negative and zero only at $c = \\mu$.\n", "\n", "**Step 3.** This is the seed of least squares: regression generalises \"predict with the mean\" to \"predict with a function of $X$\" (Day 6).\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · An average treatment effect\n", "\n", "A (hypothetical) table shows both potential outcomes for four workers:\n", "\n", "| Worker | $Y(1)$ | $Y(0)$ |\n", "|---|---|---|\n", "| 1 | 10 | 7 |\n", "| 2 | 8 | 8 |\n", "| 3 | 12 | 9 |\n", "| 4 | 6 | 4 |\n", "\n", "Compute each individual effect, the ATE, and check that $\\text{ATE} = E[Y(1)] - E[Y(0)]$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Individual effects $Y(1) - Y(0)$: $10 - 7 = 3$, $8 - 8 = 0$, $12 - 9 = 3$, $6 - 4 = 2$.\n", "\n", "**Step 2.** ATE = average of the individual effects: $(3 + 0 + 3 + 2)/4 = 2$.\n", "\n", "**Step 3.** Means of each potential outcome: $E[Y(1)] = (10 + 8 + 12 + 6)/4 = 9$ and $E[Y(0)] = (7 + 8 + 9 + 4)/4 = 7$.\n", "\n", "**Step 4.** Difference of means: $9 - 7 = 2$. ✓\n", "\n", "**Answer:** ATE = 2 either way, as linearity of expectation promises.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · How much to treat\n", "\n", "A pilot cash-transfer programme assigns each eligible household to treatment with probability $p$, so $T \\sim \\text{Bernoulli}(p)$. Compute $\\operatorname{Var}(T)$ when $p = 0.3$ and find the $p$ that maximises it. Why might the evaluation team care?\n", "\n", "
Show solution\n", "\n", "**Step 1.** By Exercise 9, $\\operatorname{Var}(T) = p(1 - p)$. At $p = 0.3$: $0.3 \\times 0.7 = 0.21$.\n", "\n", "**Step 2.** Maximise $g(p) = p - p^2$: $g'(p) = 1 - 2p = 0 \\Rightarrow p = 0.5$, and $g''(p) = -2 < 0$, so it is a maximum.\n", "\n", "**Step 3.** Maximum value: $0.5 \\times 0.5 = 0.25$.\n", "\n", "**Answer:** 0.21 at $p = 0.3$; maximised at $p = 0.5$ (value 0.25). More variation in treatment means more information about its effect, so a 50/50 split gives the most precise estimate for a given number of households (Day 10).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · A continuous voucher\n", "\n", "A voucher amount is uniform on $[0, 10]$ (thousand dollars), with density $f(x) = 1/10$. Compute $E[X]$ and $\\operatorname{Var}(X)$ by integration.\n", "\n", "
Show solution\n", "\n", "The density is $f(x) = 1/10$ on $[0, 10]$.\n", "\n", "**Step 1.** Mean:\n", "\n", "$$\n", "E[X] = \\int_0^{10} x\\cdot\\frac{1}{10}\\,dx = \\frac{1}{10}\\Big[\\frac{x^2}{2}\\Big]_0^{10} = \\frac{1}{10}\\cdot 50 = 5.\n", "$$\n", "\n", "**Step 2.** Second moment:\n", "\n", "$$\n", "E[X^2] = \\int_0^{10} x^2\\cdot\\frac{1}{10}\\,dx = \\frac{1}{10}\\Big[\\frac{x^3}{3}\\Big]_0^{10} = \\frac{1000}{30} \\approx 33.33.\n", "$$\n", "\n", "**Step 3.** Variance: $\\operatorname{Var}(X) = 33.33 - 5^2 = 8.33$.\n", "\n", "**Answer:** mean \\$5k and variance 8.33 (sd ≈ \\$2.9k), matching the formula $(b - a)^2/12 = 100/12$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Risk aversion and Jensen\n", "\n", "A household's income is \\$50k or \\$150k with equal probability, and its utility is $u(y) = \\ln y$. Compare $E[\\ln Y]$ with $\\ln E[Y]$ and find the certainty equivalent.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Expected utility:\n", "\n", "$$\n", "E[\\ln Y] = \\tfrac12\\ln 50 + \\tfrac12\\ln 150 \\approx \\tfrac12(3.9120 + 5.0106) = 4.4613.\n", "$$\n", "\n", "**Step 2.** Utility of the expected income: $E[Y] = 100$, so $\\ln E[Y] = \\ln 100 \\approx 4.6052$.\n", "\n", "**Step 3.** Compare: $4.4613 < 4.6052$, as Jensen's inequality predicts for the concave $\\ln$.\n", "\n", "**Step 4.** The certainty equivalent $c$ gives the same utility as the gamble: $\\ln c = 4.4613$, so\n", "\n", "$$\n", "c = e^{4.4613} = \\sqrt{50 \\times 150} \\approx 86.6.\n", "$$\n", "\n", "**Answer:** the household values the gamble like a sure \\$86.6k, although it is worth \\$100k on average. The \\$13.4k gap is the risk premium.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Effect heterogeneity is not identified\n", "\n", "For the four workers in Exercise 11, compute $\\operatorname{Var}(Y(1) - Y(0))$. Could you compute it from real data, where each worker shows only one potential outcome?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Individual effects from Exercise 11: 3, 0, 3, 2, with mean 2.\n", "\n", "**Step 2.** Deviations from the mean: 1, −2, 1, 0; squared: 1, 4, 1, 0.\n", "\n", "**Step 3.** Population variance: $(1 + 4 + 1 + 0)/4 = 1.5$.\n", "\n", "**Step 4.** Could real data give this? Expand the variance of a difference:\n", "\n", "$$\n", "\\operatorname{Var}(Y(1) - Y(0)) = \\operatorname{Var}(Y(1)) + \\operatorname{Var}(Y(0)) - 2\\operatorname{Cov}(Y(1), Y(0)).\n", "$$\n", "\n", "The two variances can be estimated from the treated and control groups, but the covariance needs both outcomes for the same worker, which we never see.\n", "\n", "**Answer:** 1.5 in this table, but in real data the variance of effects is **not identified**. Means are identified by linearity; the spread of effects is not.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "x = np.array([0, 500, 2000]); p = np.array([0.6, 0.3, 0.1])\n", "ex, ex2 = (x * p).sum(), (x**2 * p).sum()\n", "print(\"E[X] =\", ex, \" E[X^2] =\", ex2, \" Var =\", ex2 - ex**2)\n", "\n", "y1 = np.array([10, 8, 12, 6]); y0 = np.array([7, 8, 9, 4])\n", "print(\"ATE =\", (y1 - y0).mean(), \" Var of effects =\", (y1 - y0).var())\n", "\n", "# Jensen: average of logs < log of average\n", "inc = np.array([50, 150])\n", "print(\"E[ln Y] =\", np.log(inc).mean().round(4), \" ln E[Y] =\", np.log(inc.mean()).round(4),\n", " \" certainty equivalent =\", np.exp(np.log(inc).mean()).round(2))\n", "\n", "# Linearity check by simulation: Y = 15X - 100 with Var(X) = 400\n", "rng = np.random.default_rng(0)\n", "X = rng.normal(1200, 20, 1_000_000); Y = 15 * X - 100\n", "print(\"E[Y] ~\", Y.mean().round(1), \" Var(Y) ~\", Y.var().round(0))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Expectation is linear for any random variables; variance scales with the square of a multiplier and ignores shifts.\n", "- The ATE is a difference of two means, so it can be identified even though no individual effect is observed.\n", "- Non-linear functions do not commute with expectation (Jensen)." ] } ] }