{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 8: Estimators: Bias, Variance, MSE & Consistency\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "How to judge an estimator, and why \"unbiased\" is not the same as \"good\".\n", "\n", "**Learning outcomes**\n", "\n", "- Compute the bias, variance and mean squared error (MSE) of an estimator.\n", "- Prove the bias–variance decomposition and that the n − 1 sample variance is unbiased.\n", "- Distinguish unbiasedness from consistency and show when MSE → 0 implies consistency.\n", "- Combine independent estimates with inverse-variance weights." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Estimands, estimators, estimates\n", "\n", "- The **estimand** $\\theta$ is the quantity you want (the ATE, a population mean).\n", "- An **estimator** $\\hat\\theta$ is a rule that turns data into a number. Because data are random, $\\hat\\theta$ is a random variable with a **sampling distribution**.\n", "- An **estimate** is the number you get from your particular sample.\n", "\n", "### Three criteria\n", "\n", "$$\n", "\\operatorname{Bias}(\\hat\\theta) = E[\\hat\\theta] - \\theta, \\qquad\n", "\\operatorname{Var}(\\hat\\theta) = E\\big[(\\hat\\theta - E[\\hat\\theta])^2\\big], \\qquad\n", "\\operatorname{MSE}(\\hat\\theta) = E[(\\hat\\theta - \\theta)^2] = \\operatorname{Var}(\\hat\\theta) + \\operatorname{Bias}^2.\n", "$$\n", "\n", "A slightly biased estimator with much lower variance can have lower MSE: this **bias–variance trade-off** is behind shrinkage, ridge regression and machine-learning nuisance models (Phase 6).\n", "\n", "### Consistency\n", "\n", "$\\hat\\theta_n$ is **consistent** if $\\hat\\theta_n \\xrightarrow{p} \\theta$. A sufficient condition is $\\operatorname{MSE}(\\hat\\theta_n) \\to 0$. Consistent estimators can be biased in finite samples (e.g. $\\bar X + 1/n$); unbiased estimators can be inconsistent (e.g. using only the first observation).\n", "\n", "Most estimators in causal inference are **$\\sqrt n$-consistent and asymptotically normal**:\n", "\n", "$$\n", "\\sqrt n(\\hat\\theta_n - \\theta) \\xrightarrow{d} N(0, V),\n", "$$\n", "\n", "so their standard error is approximately $\\sqrt{V/n}$.\n", "\n", "### The sample variance\n", "\n", "The plug-in variance $\\hat\\sigma^2 = \\frac1n\\sum(X_i - \\bar X)^2$ is biased downward; the sample variance $s^2 = \\frac{1}{n-1}\\sum(X_i - \\bar X)^2$ is unbiased. The \"lost\" degree of freedom is the one used to estimate $\\bar X$.\n", "\n", "> **Causal link.** A biased estimator does not improve with more data: a confounded comparison converges to the wrong number. Identification (is the estimand recoverable at all?) comes before estimation (how precisely?).\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Two unbiased estimators\n", "\n", "From iid data with mean $\\mu$ and variance $\\sigma^2$, compare $\\hat\\theta_1 = \\bar X$ and $\\hat\\theta_2 = (X_1 + X_n)/2$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Both are unbiased: $E[\\hat\\theta_2] = (\\mu + \\mu)/2 = \\mu$.\n", "\n", "**Step 2.** But $\\operatorname{Var}(\\hat\\theta_1) = \\sigma^2/n$ while $\\operatorname{Var}(\\hat\\theta_2) = \\sigma^2/2$.\n", "\n", "**Step 3.** For $n > 2$, $\\bar X$ is more precise; $\\hat\\theta_2$ throws data away.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · MSE arithmetic\n", "\n", "An estimator has bias 2 and variance 5. What is its MSE and root-MSE?\n", "\n", "
Show solution\n", "\n", "$\\operatorname{MSE} = 5 + 2^2 = 9$, root-MSE $= 3$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · A shrinkage estimator\n", "\n", "Let $\\hat\\theta_c = c\\,\\bar X$ with $\\sigma^2 = 100$, $n = 25$ and $c = 0.9$. Compute its MSE when $\\mu = 10$ and when $\\mu = 2$, and compare with $\\bar X$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Bias of $c\\bar X$: $E[c\\bar X] - \\mu = (c - 1)\\mu$. Variance: $c^2\\sigma^2/n = 0.81 \\times 100/25 = 3.24$.\n", "\n", "**Step 2.** For $\\bar X$ itself: bias 0, variance $100/25 = 4$, so MSE $= 4$.\n", "\n", "**Step 3.** If $\\mu = 10$: bias $= -0.1 \\times 10 = -1$, MSE $= 3.24 + 1 = 4.24 > 4$.\n", "\n", "**Step 4.** If $\\mu = 2$: bias $= -0.2$, MSE $= 3.24 + 0.04 = 3.28 < 4$.\n", "\n", "**Answer:** shrinking toward zero helps when the truth is near zero (MSE 3.28 versus 4) and hurts when it is far away (4.24 versus 4).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · The plug-in variance\n", "\n", "For $n = 10$ and $\\sigma^2 = 50$, what is $E[\\hat\\sigma^2]$ for the plug-in (divide-by-$n$) variance?\n", "\n", "
Show solution\n", "\n", "$E[\\hat\\sigma^2] = \\frac{n - 1}{n}\\sigma^2 = 0.9 \\times 50 = 45$: a bias of $-5$ (Exercise 7 proves this).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Biased but consistent\n", "\n", "Is $\\tilde\\theta_n = \\bar X_n + 1/n$ unbiased? Consistent?\n", "\n", "
Show solution\n", "\n", "Biased: $E[\\tilde\\theta_n] = \\mu + 1/n$. Consistent: bias $\\to 0$ and variance $\\sigma^2/n \\to 0$, so the MSE $\\to 0$.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Bias–variance decomposition\n", "\n", "Prove $\\operatorname{MSE}(\\hat\\theta) = \\operatorname{Var}(\\hat\\theta) + \\operatorname{Bias}(\\hat\\theta)^2$.\n", "\n", "
Show solution\n", "\n", "Add and subtract $E[\\hat\\theta]$:\n", "\n", "$$\n", "E[(\\hat\\theta - \\theta)^2] = E[(\\hat\\theta - E\\hat\\theta)^2] + 2(E\\hat\\theta - \\theta)\\underbrace{E[\\hat\\theta - E\\hat\\theta]}_{0} + (E\\hat\\theta - \\theta)^2.\n", "$$\n", "\n", "This is the same algebra as Day 3, Exercise 10.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Why divide by n − 1\n", "\n", "Prove $E\\big[\\sum_{i=1}^n(X_i - \\bar X)^2\\big] = (n - 1)\\sigma^2$ for iid data.\n", "\n", "
Show solution\n", "\n", "Use $\\sum(X_i - \\bar X)^2 = \\sum(X_i - \\mu)^2 - n(\\bar X - \\mu)^2$ (expand $X_i - \\mu = (X_i - \\bar X) + (\\bar X - \\mu)$; the cross term sums to zero). Taking expectations:\n", "\n", "$$\n", "E\\Big[\\sum(X_i - \\bar X)^2\\Big] = n\\sigma^2 - n\\cdot\\frac{\\sigma^2}{n} = (n - 1)\\sigma^2.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · MSE → 0 implies consistency\n", "\n", "Prove that if $\\operatorname{MSE}(\\hat\\theta_n) \\to 0$ then $\\hat\\theta_n \\xrightarrow{p} \\theta$.\n", "\n", "
Show solution\n", "\n", "Apply Markov's inequality to $(\\hat\\theta_n - \\theta)^2$:\n", "\n", "$$\n", "P(|\\hat\\theta_n - \\theta| \\ge \\varepsilon) \\le \\frac{E[(\\hat\\theta_n - \\theta)^2]}{\\varepsilon^2} = \\frac{\\operatorname{MSE}(\\hat\\theta_n)}{\\varepsilon^2} \\to 0.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · The sample mean is the best linear unbiased estimator\n", "\n", "Among estimators $\\sum_i w_iX_i$ with $\\sum_i w_i = 1$ (so they are unbiased), show that equal weights minimise the variance.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The variance is $\\sigma^2\\sum w_i^2$.\n", "\n", "**Step 2.** Write $w_i = \\frac1n + d_i$ with $\\sum d_i = 0$.\n", "\n", "**Step 3.** Then\n", "\n", "$$\n", "\\sum w_i^2 = \\frac1n + \\frac2n\\sum d_i + \\sum d_i^2 = \\frac1n + \\sum d_i^2 \\ge \\frac1n,\n", "$$\n", "\n", "**Step 4.** with equality only when all $d_i = 0$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · The best shrinkage factor\n", "\n", "For $\\hat\\theta_c = c\\bar X$, find the $c$ that minimises the MSE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{MSE}(c) = c^2\\sigma^2/n + (c - 1)^2\\mu^2$.\n", "\n", "**Step 2.** Setting the derivative to zero, $2c\\sigma^2/n + 2(c - 1)\\mu^2 = 0$, gives\n", "\n", "$$\n", "c^* = \\frac{\\mu^2}{\\mu^2 + \\sigma^2/n}.\n", "$$\n", "\n", "**Step 3.** It depends on the unknown $\\mu$, which is why practical shrinkage (empirical Bayes, Day 13) estimates it from the data.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · The difference in means is unbiased in a cash-transfer RCT\n", "\n", "Households are randomly assigned to receive a cash transfer ($T = 1$) or not, independently of their potential consumption levels $(Y(1), Y(0))$. Show that the difference between the treated and control groups' mean consumption, $\\bar Y_1 - \\bar Y_0$, is unbiased for the ATE.\n", "\n", "
Show solution\n", "\n", "**Step 1.** In the treated group each observed $Y_i = Y_i(1)$, so $E[\\bar Y_1] = E[Y(1) \\mid T = 1]$.\n", "\n", "**Step 2.** Random assignment gives $E[Y(1) \\mid T = 1] = E[Y(1)]$ (Day 6, Exercise 14).\n", "\n", "**Step 3.** Likewise $E[\\bar Y_0] = E[Y(0) \\mid T = 0] = E[Y(0)]$.\n", "\n", "**Step 4.** Linearity: $E[\\bar Y_1 - \\bar Y_0] = E[Y(1)] - E[Y(0)] = \\text{ATE}$. $\\blacksquare$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Pooling two evaluations\n", "\n", "Two independent RCTs of the same school-meals programme estimate its effect on attendance as 2.0 days (SE 0.5) and 3.0 days (SE 1.0). Find the inverse-variance weighted estimate and its standard error.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Weights are inverse variances: $w_1 = 1/0.5^2 = 4$ and $w_2 = 1/1.0^2 = 1$.\n", "\n", "**Step 2.** Pooled estimate:\n", "\n", "$$\n", "\\hat\\theta = \\frac{4(2.0) + 1(3.0)}{4 + 1} = \\frac{11}{5} = 2.2.\n", "$$\n", "\n", "**Step 3.** Variance of the pooled estimate: $1/(w_1 + w_2) = 1/5$, so $\\operatorname{SE} = 1/\\sqrt5 \\approx 0.447$.\n", "\n", "**Answer:** 2.2 days (SE 0.45), more precise than either study alone. This is fixed-effect meta-analysis.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · More data does not cure confounding\n", "\n", "In Day 4's confounded DGP the naive difference converges to 17 while the true effect is 5. Is the naive estimator consistent for the ATE?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The naive difference converges to its own probability limit, $E[Y \\mid T = 1] - E[Y \\mid T = 0] = 17$.\n", "\n", "**Step 2.** Its bias for the ATE is $17 - 5 = 12$ at every sample size.\n", "\n", "**Step 3.** Its variance shrinks to zero, so its MSE tends to $0 + 12^2 = 144$, not zero.\n", "\n", "**Answer:** not consistent for the ATE. More data gives an ever more precise estimate of the wrong number.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Mean or median test score?\n", "\n", "A ministry summarises the test scores of 100 pupils per school. Scores are roughly normal with $\\sigma = 10$. The sample median has variance about $\\pi\\sigma^2/(2n)$. Compare its precision with the sample mean's, and say when the median might be preferred.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Variance of the mean: $\\sigma^2/n = 100/100 = 1$.\n", "\n", "**Step 2.** Variance of the median: $\\pi\\sigma^2/(2n) = \\pi/2 \\approx 1.571$.\n", "\n", "**Step 3.** Relative efficiency: $1/1.571 = 2/\\pi \\approx 0.64$; the median needs about 57% more pupils for the same precision.\n", "\n", "**Answer:** with normal scores the mean is more precise. With heavy tails or data errors (e.g. a few mis-coded scores of 0) the median can win; try the t-distribution in the Lab.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Planning a consumption survey\n", "\n", "An unbiased estimator of mean monthly household consumption (in \\$100) has variance $4/n$. How many households are needed for its MSE to be below 0.01, i.e. a root-MSE below \\$10?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Unbiased, so MSE = variance = $4/n$.\n", "\n", "**Step 2.** Require $4/n < 0.01$, i.e. $n > 4/0.01 = 400$.\n", "\n", "**Answer:** more than 400 households.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "rng = np.random.default_rng(8)\n", "reps, n, sigma = 100_000, 25, 10\n", "for mu in (10, 2):\n", " xbar = rng.normal(mu, sigma, (reps, n)).mean(1)\n", " for name, est in ((\"mean\", xbar), (\"0.9*mean\", 0.9 * xbar)):\n", " print(f\"mu={mu:>2} {name:>8}: bias {est.mean() - mu:+.3f} MSE {((est - mu)**2).mean():.3f}\")\n", "\n", "# The n vs n-1 variance\n", "x = rng.normal(0, np.sqrt(50), (100_000, 10))\n", "print(\"E[plug-in var] ~\", x.var(axis=1, ddof=0).mean().round(2), \"; E[s^2] ~\", x.var(axis=1, ddof=1).mean().round(2))\n", "\n", "# Mean vs median efficiency with normal data\n", "x = rng.normal(0, 10, (50_000, 100))\n", "print(\"Var(mean) ~\", x.mean(1).var().round(3), \" Var(median) ~\", np.median(x, 1).var().round(3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- MSE = variance + bias²: judge estimators on both.\n", "- Consistency is about the limit, unbiasedness about the average: neither implies the other.\n", "- No sample size fixes a biased (unidentified) comparison." ] } ] }