{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 9: Confidence Intervals & the Bootstrap\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Turning a standard error into an interval with a known coverage rate, analytically or by resampling.\n", "\n", "**Learning outcomes**\n", "\n", "- Build normal and t confidence intervals for means, proportions and differences in means.\n", "- Explain what \"95% coverage\" does and does not mean.\n", "- Describe the nonparametric bootstrap and compute a percentile interval.\n", "- Read standard errors and significance off a published interval." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The pivot\n", "\n", "If $\\hat\\theta \\approx N(\\theta, \\operatorname{SE}^2)$, then $(\\hat\\theta - \\theta)/\\operatorname{SE} \\approx N(0, 1)$, whose distribution does not depend on $\\theta$ (a **pivot**). So\n", "\n", "$$\n", "P\\big(-z_{1-\\alpha/2} \\le \\tfrac{\\hat\\theta - \\theta}{\\operatorname{SE}} \\le z_{1-\\alpha/2}\\big) \\approx 1 - \\alpha\n", "\\;\\Longrightarrow\\;\n", "\\text{CI}_{1-\\alpha} = \\hat\\theta \\pm z_{1-\\alpha/2}\\,\\operatorname{SE}.\n", "$$\n", "\n", "Common critical values: $z_{0.95} = 1.645$ (90%), $z_{0.975} = 1.96$ (95%), $z_{0.995} = 2.576$ (99%).\n", "\n", "With small samples from a normal population and an estimated SD, replace $z$ by the Student $t$ quantile with $n - 1$ degrees of freedom.\n", "\n", "### Interpretation\n", "\n", "The **interval** is random; $\\theta$ is fixed. \"95% confidence\" means that the *procedure* captures $\\theta$ in 95% of repeated samples. It does **not** mean there is a 95% probability that $\\theta$ lies in the interval you computed (that is a Bayesian credible interval, Day 13). The CI coverage simulator in the Lab makes this concrete.\n", "\n", "### Standard errors you will use constantly\n", "\n", "| Estimator | Standard error |\n", "|---|---|\n", "| Mean | $s/\\sqrt n$ |\n", "| Proportion | $\\sqrt{\\hat p(1 - \\hat p)/n}$ |\n", "| Difference in means (independent groups) | $\\sqrt{s_1^2/n_1 + s_0^2/n_0}$ |\n", "\n", "### The nonparametric bootstrap\n", "\n", "When the SE formula is unknown or messy (medians, ratios, complex estimators):\n", "\n", "1. Draw $n$ observations **with replacement** from your sample.\n", "2. Compute the estimator on this resample, $\\hat\\theta^{*}_b$.\n", "3. Repeat $B$ times (say $B = 2{,}000$).\n", "4. Use the standard deviation of the $\\hat\\theta^{*}_b$ as the SE, or their 2.5th and 97.5th percentiles as a **percentile interval**.\n", "\n", "The bootstrap treats the sample as a stand-in for the population. Resample in the way the data were generated: with clustered data, resample whole clusters.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · A 95% interval for a mean\n", "\n", "$\\bar X = 52$, $s = 10$, $n = 100$. Build a 95% CI.\n", "\n", "
Show solution\n", "\n", "$\\operatorname{SE} = 1$, so $52 \\pm 1.96 = [50.04,\\; 53.96]$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Raising the confidence level\n", "\n", "Build the 99% CI for the same data. Why is it wider?\n", "\n", "
Show solution\n", "\n", "$52 \\pm 2.576 = [49.42,\\; 54.58]$. More confidence requires a wider net.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · A proportion\n", "\n", "In a survey of 600, 42% support a policy. Build a 95% CI.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{SE} = \\sqrt{\\hat p(1 - \\hat p)/n} = \\sqrt{0.42 \\times 0.58/600} = \\sqrt{0.000406} \\approx 0.0201$.\n", "\n", "**Step 2.** Margin of error: $1.96 \\times 0.0201 \\approx 0.0395$.\n", "\n", "**Step 3.** Interval: $0.42 \\pm 0.0395$.\n", "\n", "**Answer:** $[0.381,\\; 0.459]$: support is between about 38% and 46%.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · A treatment effect\n", "\n", "Treated: mean 78, sd 12, $n = 144$. Control: mean 74, sd 15, $n = 225$. Build a 95% CI for the difference.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Point estimate: $78 - 74 = 4$.\n", "\n", "**Step 2.** Standard error with independent groups:\n", "\n", "$$\n", "\\operatorname{SE} = \\sqrt{\\frac{12^2}{144} + \\frac{15^2}{225}} = \\sqrt{1 + 1} \\approx 1.414.\n", "$$\n", "\n", "**Step 3.** Margin: $1.96 \\times 1.414 \\approx 2.77$.\n", "\n", "**Answer:** $4 \\pm 2.77 = [1.23,\\; 6.77]$; zero is excluded, so the effect is significant at 5%.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · A small sample\n", "\n", "$n = 16$, $\\bar X = 10$, $s = 4$. Build a 95% t interval ($t_{15, 0.975} = 2.131$).\n", "\n", "
Show solution\n", "\n", "$\\operatorname{SE} = 4/4 = 1$, so $10 \\pm 2.131 = [7.87,\\; 12.13]$, slightly wider than the normal interval $10 \\pm 1.96$.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Coverage of the known-σ interval\n", "\n", "If $X_i \\sim N(\\mu, \\sigma^2)$ iid with known $\\sigma$, prove that $\\bar X \\pm 1.96\\,\\sigma/\\sqrt n$ contains $\\mu$ with probability exactly 0.95.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $Z = (\\bar X - \\mu)/(\\sigma/\\sqrt n) \\sim N(0, 1)$ exactly.\n", "\n", "**Step 2.** The event $\\mu \\in [\\bar X - 1.96\\,\\sigma/\\sqrt n,\\; \\bar X + 1.96\\,\\sigma/\\sqrt n]$ is the same event as $|Z| \\le 1.96$, whose probability is $\\Phi(1.96) - \\Phi(-1.96) = 0.95$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Width and sample size\n", "\n", "Show that the width of a CI is proportional to $1/\\sqrt n$, and find the sample size needed to halve it.\n", "\n", "
Show solution\n", "\n", "Width $= 2z\\,\\sigma/\\sqrt n$. Replacing $n$ by $kn$ multiplies it by $1/\\sqrt k$; halving needs $k = 4$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Left out of the bootstrap\n", "\n", "Show that the probability a given observation does not appear in a bootstrap resample is $(1 - 1/n)^n$, and find its limit.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Each of the $n$ independent draws misses the observation with probability $1 - 1/n$, so it is never drawn with probability $(1 - 1/n)^n$.\n", "\n", "**Step 2.** Since $\\ln(1 - 1/n)^n = n\\ln(1 - 1/n) \\to -1$, the limit is $e^{-1} \\approx 0.368$.\n", "\n", "**Step 3.** Each resample contains about 63.2% of the distinct observations.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Duality of tests and intervals\n", "\n", "Show that a two-sided 5% test of $H_0: \\theta = \\theta_0$ rejects exactly when $\\theta_0$ lies outside the 95% CI.\n", "\n", "
Show solution\n", "\n", "**Step 1.** The test rejects when $|\\hat\\theta - \\theta_0|/\\operatorname{SE} > 1.96$, i.e. when $\\theta_0 < \\hat\\theta - 1.96\\operatorname{SE}$ or $\\theta_0 > \\hat\\theta + 1.96\\operatorname{SE}$, which is exactly \"outside the CI\".\n", "\n", "**Step 2.** A CI is the set of null values you would not reject.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Reverse-engineering a paper\n", "\n", "A paper reports an effect with 95% CI $[1.2,\\; 5.6]$. Recover the estimate, its SE and its z-statistic.\n", "\n", "
Show solution\n", "\n", "**Step 1.** A 95% CI is symmetric around the estimate: $\\hat\\theta = (1.2 + 5.6)/2 = 3.4$.\n", "\n", "**Step 2.** Its half-width is $1.96 \\times \\operatorname{SE}$: $\\operatorname{SE} = (5.6 - 1.2)/(2 \\times 1.96) = 4.4/3.92 \\approx 1.12$.\n", "\n", "**Step 3.** $z = 3.4/1.12 \\approx 3.03$, so $p = 2(1 - \\Phi(3.03)) \\approx 0.002$.\n", "\n", "**Answer:** estimate 3.4, SE 1.12, $z \\approx 3.0$, $p \\approx 0.002$.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Clustered data\n", "\n", "Using Day 7's design effect of 1.95, by how much should a naive CI be widened?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Clustering multiplies the variance by the design effect, 1.95, so the SE must be multiplied by $\\sqrt{1.95} \\approx 1.40$.\n", "\n", "**Step 2.** The naive interval uses $\\pm1.96$ naive SEs, which is only $1.96/1.40 = 1.40$ correct SEs.\n", "\n", "**Step 3.** Its true coverage is $2\\Phi(1.40) - 1 \\approx 0.84$.\n", "\n", "**Answer:** widen it by 40%; otherwise the \"95%\" interval covers only about 84% of the time.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · A percentile interval for median rent\n", "\n", "A housing agency bootstraps the median monthly rent in a city from a sample of 300 listings, computes 1,000 bootstrap medians and sorts them. Which sorted values form the 95% percentile interval, and how would you report it?\n", "\n", "
Show solution\n", "\n", "**Step 1.** A 95% interval leaves 2.5% in each tail.\n", "\n", "**Step 2.** 2.5% of 1,000 is 25 and 97.5% is 975, so take roughly the 25th and 975th sorted bootstrap medians (`numpy.percentile` interpolates between neighbours).\n", "\n", "**Answer:** report \"median rent \\$X (95% bootstrap CI: [25th value, 975th value])\". The interval need not be symmetric around \\$X, which is useful for skewed rents.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Bootstrapping a median\n", "\n", "Health costs are heavily right-skewed. Why bootstrap the median rather than use a formula, and what are the steps?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Why not a formula: the median's analytic SE depends on the density of costs at the median, which is hard to estimate with skewed data.\n", "\n", "**Step 2.** Draw 300 patients *with replacement* from the sample and compute the median cost.\n", "\n", "**Step 3.** Repeat 2,000 times.\n", "\n", "**Step 4.** Report the SD of the 2,000 medians as the SE, or their 2.5th and 97.5th percentiles as a 95% interval.\n", "\n", "**Answer:** the bootstrap needs no formula, only the ability to recompute the median.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Overlapping intervals for two regions\n", "\n", "A labour-force survey estimates mean hourly wages of \\$10 (SE 1) in region A and \\$13 (SE 1) in region B, from independent samples. Their 95% CIs overlap. Is the difference significant?\n", "\n", "
Show solution\n", "\n", "**Step 1.** The CIs: A $10 \\pm 1.96 = [8.04, 11.96]$; B $13 \\pm 1.96 = [11.04, 14.96]$. They overlap on $[11.04, 11.96]$.\n", "\n", "**Step 2.** Test the difference directly: $13 - 10 = 3$ with $\\operatorname{SE} = \\sqrt{1^2 + 1^2} \\approx 1.41$.\n", "\n", "**Step 3.** $z = 3/1.41 \\approx 2.12$, $p \\approx 0.034$.\n", "\n", "**Answer:** significant at 5% despite the overlap. Overlapping intervals are not a test; always test the difference.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · What the bootstrap SE of mean income estimates\n", "\n", "You bootstrap the mean income of $n$ surveyed households. Show that, if you could average over *all* possible resamples, the bootstrap variance of the mean equals $\\hat\\sigma^2/n$, where $\\hat\\sigma^2 = \\frac1n\\sum(X_i - \\bar X)^2$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Each draw in a resample picks one of the $n$ households with equal probability $1/n$.\n", "\n", "**Step 2.** So one draw has mean $\\frac1n\\sum X_i = \\bar X$ and variance $\\frac1n\\sum(X_i - \\bar X)^2 = \\hat\\sigma^2$.\n", "\n", "**Step 3.** A resample mean averages $n$ independent such draws, so its variance is $\\hat\\sigma^2/n$.\n", "\n", "**Answer:** the bootstrap variance is $\\hat\\sigma^2/n$, essentially the textbook $s^2/n$ (smaller by the factor $(n - 1)/n$).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from scipy import stats\n", "\n", "rng = np.random.default_rng(9)\n", "\n", "# Coverage of the 95% t interval with skewed data (log-normal, true mean known)\n", "mu_true = np.exp(0.5)\n", "reps, n = 20_000, 30\n", "x = rng.lognormal(0, 1, (reps, n))\n", "m, s = x.mean(1), x.std(1, ddof=1)\n", "t = stats.t.ppf(0.975, n - 1)\n", "cover = np.mean((m - t * s / np.sqrt(n) <= mu_true) & (mu_true <= m + t * s / np.sqrt(n)))\n", "print(f\"coverage with skewed data, n=30: {cover:.3f} (nominal 0.95)\")\n", "# With strongly skewed data and n = 30 the CLT has not fully kicked in: coverage falls short.\n", "\n", "# Bootstrap the median of one sample\n", "sample = rng.lognormal(8, 1, 200) # health costs\n", "boots = np.array([np.median(rng.choice(sample, sample.size)) for _ in range(2000)])\n", "print(\"median:\", round(np.median(sample)), \" bootstrap SE:\", round(boots.std(ddof=1)),\n", " \" 95% percentile CI:\", np.percentile(boots, [2.5, 97.5]).round())\n", "\n", "print(\"P(left out):\", round((1 - 1/200)**200, 4), \" vs e^-1 =\", round(np.exp(-1), 4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- $\\hat\\theta \\pm 1.96\\,\\operatorname{SE}$ covers $\\theta$ in about 95% of repeated samples.\n", "- The bootstrap estimates a sampling distribution by resampling your data the way they were collected.\n", "- Compare estimates by testing their difference, not by checking whether intervals overlap." ] } ] }