{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 10: Hypothesis Testing, p-Values & Power\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Deciding whether an effect is distinguishable from zero, and designing studies that can tell.\n", "\n", "**Learning outcomes**\n", "\n", "- Set up a test, compute a z-statistic and p-value, and state the decision correctly.\n", "- Prove that p-values are uniform under the null.\n", "- Derive the power formula, the required sample size and the minimum detectable effect.\n", "- Explain multiple testing, statistical versus economic significance, and the winner's curse." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### The logic of a test\n", "\n", "1. State a **null hypothesis** $H_0: \\tau = 0$ and an **alternative** $H_1: \\tau \\ne 0$.\n", "2. Compute a **test statistic** whose distribution under $H_0$ is known: $z = \\hat\\tau/\\operatorname{SE} \\approx N(0, 1)$.\n", "3. The **p-value** is the probability, *if $H_0$ were true*, of a statistic at least as extreme as the one observed: $p = 2\\big(1 - \\Phi(|z|)\\big)$.\n", "4. Reject $H_0$ at level $\\alpha$ when $p < \\alpha$, equivalently $|z| > z_{1-\\alpha/2}$.\n", "\n", "### Two kinds of error\n", "\n", "| | $H_0$ true | $H_0$ false |\n", "|---|---|---|\n", "| Reject | Type I error (prob. $\\alpha$) | Correct (prob. = **power**) |\n", "| Do not reject | Correct | Type II error (prob. $\\beta$) |\n", "\n", "### Power, sample size and the MDE\n", "\n", "If the true effect is $\\tau$, then $z \\approx N(\\tau/\\operatorname{SE}, 1)$ and\n", "\n", "$$\n", "\\text{Power} \\approx \\Phi\\Big(\\frac{|\\tau|}{\\operatorname{SE}} - z_{1-\\alpha/2}\\Big).\n", "$$\n", "\n", "For two arms of size $n$ with outcome SD $\\sigma$, $\\operatorname{SE} = \\sigma\\sqrt{2/n}$. Solving for power $1 - \\beta$:\n", "\n", "$$\n", "n = \\frac{2\\sigma^2\\,(z_{1-\\alpha/2} + z_{1-\\beta})^2}{\\tau^2}, \\qquad\n", "\\text{MDE} = (z_{1-\\alpha/2} + z_{1-\\beta})\\,\\operatorname{SE}.\n", "$$\n", "\n", "With $\\alpha = 0.05$ and 80% power, $z_{0.975} + z_{0.8} = 1.96 + 0.84 = 2.80$: the **MDE is 2.8 standard errors**.\n", "\n", "### Pitfalls\n", "\n", "- **Multiple testing:** with many tests, some will be \"significant\" by chance.\n", "- **Statistical ≠ economic significance:** huge samples make tiny effects significant.\n", "- **Winner's curse:** in underpowered studies, estimates that clear the significance bar are exaggerated.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Hypotheses\n", "\n", "A job-training programme's effect on earnings is $\\tau$. Write the null and two-sided alternative for \"the programme has an effect\".\n", "\n", "
Show solution\n", "\n", "$H_0: \\tau = 0$ versus $H_1: \\tau \\ne 0$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · A z-statistic\n", "\n", "The estimated effect is 3.2 (thousand dollars) with SE 1.25. Compute $z$.\n", "\n", "
Show solution\n", "\n", "$z = 3.2/1.25 = 2.56$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Decision and p-value\n", "\n", "Is the effect significant at 5%? Compute the p-value.\n", "\n", "
Show solution\n", "\n", "$|z| = 2.56 > 1.96$, so reject $H_0$. $p = 2(1 - \\Phi(2.56)) \\approx 0.0105$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · What a p-value is not\n", "\n", "A report says \"p = 0.01, so there is a 1% chance the programme has no effect.\" Correct it.\n", "\n", "
Show solution\n", "\n", "The p-value is computed *assuming* $H_0$ is true: if the programme had no effect, results at least this extreme would occur about 1% of the time. It is not $P(H_0 \\mid \\text{data})$; that requires a prior (Day 13).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Trading off errors\n", "\n", "What happens to Type I and Type II error rates if you lower $\\alpha$ from 0.05 to 0.01 with the same data?\n", "\n", "
Show solution\n", "\n", "Type I falls to 1%, but the critical value rises from 1.96 to 2.576, so real effects are rejected less often: Type II error rises and power falls. Only more information (larger $n$, less noise) lowers both.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · p-values are uniform under the null\n", "\n", "Let the statistic $Z$ have continuous CDF $F$ under $H_0$, and define the one-sided p-value $p = 1 - F(Z)$. Show $p \\sim \\text{Uniform}(0, 1)$ under $H_0$.\n", "\n", "
Show solution\n", "\n", "For $u \\in (0, 1)$: $P(p \\le u) = P(F(Z) \\ge 1 - u) = P(Z \\ge F^{-1}(1 - u)) = 1 - F(F^{-1}(1 - u)) = u$. So under $H_0$ every p-value is equally likely, and $P(p < 0.05) = 0.05$ by construction.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · The power formula\n", "\n", "Derive $\\text{Power} = \\Phi(\\tau/\\operatorname{SE} - z_{1-\\alpha/2}) + \\Phi(-\\tau/\\operatorname{SE} - z_{1-\\alpha/2})$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Under the alternative, $z = \\hat\\tau/\\operatorname{SE} \\sim N(\\delta, 1)$ with $\\delta = \\tau/\\operatorname{SE}$.\n", "\n", "**Step 2.** With $c = z_{1-\\alpha/2}$:\n", "\n", "$$\n", "P(z > c) = 1 - \\Phi(c - \\delta) = \\Phi(\\delta - c), \\qquad P(z < -c) = \\Phi(-c - \\delta).\n", "$$\n", "\n", "**Step 3.** Add them.\n", "\n", "**Step 4.** For $\\delta > 0$ the second term is tiny, giving the approximation in the theory block.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Sample size\n", "\n", "Derive $n = 2\\sigma^2(z_{1-\\alpha/2} + z_{1-\\beta})^2/\\tau^2$ per arm, and apply it with $\\sigma = 10$, $\\tau = 2$, $\\alpha = 0.05$, power 0.8.\n", "\n", "
Show solution\n", "\n", "**Derivation**\n", "\n", "**Step 1.** Approximate power: $\\Phi(\\tau/\\operatorname{SE} - z_{1-\\alpha/2}) = 1 - \\beta$.\n", "\n", "**Step 2.** Apply $\\Phi^{-1}$: $\\tau/\\operatorname{SE} = z_{1-\\alpha/2} + z_{1-\\beta}$.\n", "\n", "**Step 3.** Substitute $\\operatorname{SE} = \\sigma\\sqrt{2/n}$: $\\tau\\sqrt n/(\\sigma\\sqrt2) = z_{1-\\alpha/2} + z_{1-\\beta}$.\n", "\n", "**Step 4.** Square and solve: $n = 2\\sigma^2(z_{1-\\alpha/2} + z_{1-\\beta})^2/\\tau^2$.\n", "\n", "**Application**\n", "\n", "**Step 5.** $z_{0.975} = 1.96$, $z_{0.80} = 0.8416$, so $(1.96 + 0.8416)^2 = 2.8016^2 \\approx 7.849$.\n", "\n", "**Step 6.** $n = 2(100)(7.849)/4 \\approx 392.4$; round up.\n", "\n", "**Answer:** 393 per arm.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Unequal allocation\n", "\n", "With a fixed total $N$ and a share $p$ treated, $\\operatorname{Var}(\\hat\\tau) = \\sigma^2/(Np(1 - p))$. Show $p = 0.5$ is optimal and find the cost of a 70/30 split.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{Var}(\\hat\\tau) = \\sigma^2/(Np(1 - p))$ is smallest when $p(1 - p)$ is largest, which happens at $p = 0.5$ (Day 3, Exercise 12).\n", "\n", "**Step 2.** At a 70/30 split, $p(1 - p) = 0.21$, versus 0.25 at 50/50.\n", "\n", "**Step 3.** Variance ratio: $0.25/0.21 \\approx 1.19$.\n", "\n", "**Answer:** 50/50 is optimal; a 70/30 split needs about 19% more people for the same precision (still worth it if treatment is expensive, Day 26).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · The winner's curse\n", "\n", "The true effect is $\\tau = 1$ and $\\operatorname{SE} = 1$. Compute the power, and the expected estimate *given* that it is significantly positive. *Hint: if $X \\sim N(\\mu, 1)$, then $E[X \\mid X > c] = \\mu + \\phi(c - \\mu)/(1 - \\Phi(c - \\mu))$.*\n", "\n", "
Show solution\n", "\n", "**Step 1.** Power: $\\tau/\\operatorname{SE} = 1$, so power $\\approx \\Phi(1 - 1.96) = \\Phi(-0.96) \\approx 0.17$.\n", "\n", "**Step 2.** Given a significant positive result, use the truncated-normal mean with $\\mu = 1$, $c = 1.96$, $c - \\mu = 0.96$:\n", "\n", "$$\n", "E[\\hat\\tau \\mid \\hat\\tau > 1.96] = 1 + \\frac{\\phi(0.96)}{1 - \\Phi(0.96)}.\n", "$$\n", "\n", "**Step 3.** $\\phi(0.96) = 0.3989\\,e^{-0.96^2/2} \\approx 0.2516$ and $1 - \\Phi(0.96) \\approx 0.1685$; ratio ≈ 1.493.\n", "\n", "**Step 4.** $E[\\hat\\tau \\mid \\text{significant}] \\approx 2.49$.\n", "\n", "**Answer:** power is only 17%, and the significant estimates average 2.5 times the true effect. Small published studies often overstate effects for this reason.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · One-sided or two-sided? A cash-transfer test\n", "\n", "A conditional cash transfer's effect on school enrolment gives $z = 1.80$. The team had planned a two-sided test. Compute the one-sided and two-sided p-values. Which should be reported?\n", "\n", "
Show solution\n", "\n", "**Step 1.** One-sided: $P(Z > 1.80) = 1 - \\Phi(1.80) \\approx 0.036$.\n", "\n", "**Step 2.** Two-sided: $2 \\times 0.036 = 0.072$.\n", "\n", "**Answer:** report the pre-planned two-sided $p = 0.072$. Switching to one-sided after seeing the sign doubles the false-positive rate; one-sided tests are fine only if pre-registered for a substantive reason (e.g. the transfer cannot plausibly reduce enrolment).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · An underpowered nutrition trial\n", "\n", "A nutrition supplement trial can estimate the effect on child height with SE 1.5 cm. If the true effect is 3 cm, what is the trial's power at the 5% level?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Standardised effect: $\\tau/\\operatorname{SE} = 3/1.5 = 2$.\n", "\n", "**Step 2.** Power $\\approx \\Phi(2 - 1.96) = \\Phi(0.04) \\approx 0.52$.\n", "\n", "**Answer:** about 52%: a coin flip. A null result from this trial would say little about whether the supplement works.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Minimum detectable effect\n", "\n", "With 500 per arm and $\\sigma = 20$, what is the MDE at 5% significance and 80% power?\n", "\n", "
Show solution\n", "\n", "**Step 1.** $\\operatorname{SE} = \\sigma\\sqrt{2/n} = 20\\sqrt{2/500} = 20 \\times 0.0632 \\approx 1.265$.\n", "\n", "**Step 2.** MDE multiplier at 5% and 80% power: $1.96 + 0.84 = 2.80$.\n", "\n", "**Step 3.** $\\text{MDE} = 2.80 \\times 1.265 \\approx 3.54$.\n", "\n", "**Answer:** the study can reliably detect effects of about 3.5 units or more.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Twenty outcomes\n", "\n", "A study tests 20 independent outcomes at 5%, and the programme truly affects none. What is the chance of at least one \"significant\" result? What threshold does Bonferroni use?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Each test has a 0.95 chance of not giving a false positive; independent, so all 20 avoid one with probability $0.95^{20} \\approx 0.358$.\n", "\n", "**Step 2.** At least one false positive: $1 - 0.358 \\approx 0.64$.\n", "\n", "**Step 3.** Bonferroni divides the level by the number of tests: $0.05/20 = 0.0025$. By Boole's inequality (Day 1), the chance of any false positive is then at most $20 \\times 0.0025 = 0.05$.\n", "\n", "**Answer:** a 64% chance of at least one spurious \"effect\"; Bonferroni tests each outcome at 0.0025.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Statistical versus economic significance\n", "\n", "On 5 million tax records, a policy raises earnings by \\$1.50 a year with $p < 0.0001$. Interpret.\n", "\n", "
Show solution\n", "\n", "**Step 1.** $z = \\hat\\tau/\\operatorname{SE}$ with $\\operatorname{SE} = \\sigma/\\sqrt n$.\n", "\n", "**Step 2.** With $n = 5{,}000{,}000$, $\\sqrt n \\approx 2{,}236$, so the SE is tiny and even \\$1.50 gives a huge $z$.\n", "\n", "**Answer:** the effect is precisely estimated (statistically significant) but negligible in size (economically insignificant). Report the effect and its CI, not just the p-value.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from scipy import stats\n", "\n", "z = 3.2 / 1.25\n", "print(f\"z = {z:.2f}, p = {2 * stats.norm.sf(z):.4f}\")\n", "\n", "za, zb = stats.norm.ppf(0.975), stats.norm.ppf(0.80)\n", "print(\"n per arm:\", np.ceil(2 * 10**2 * (za + zb)**2 / 2**2))\n", "print(\"MDE:\", round((za + zb) * 20 * np.sqrt(2 / 500), 2))\n", "print(\"P(>=1 false positive in 20):\", round(1 - 0.95**20, 3))\n", "\n", "# Winner's curse by simulation\n", "rng = np.random.default_rng(10)\n", "est = rng.normal(1, 1, 1_000_000)\n", "sig = est > za\n", "print(f\"power {sig.mean():.3f}; mean significant estimate {est[sig].mean():.2f} (truth 1)\")\n", "\n", "# p-values are uniform under H0\n", "p = 2 * stats.norm.sf(np.abs(rng.normal(0, 1, 100_000)))\n", "print(\"share p < 0.05 under H0:\", round((p < 0.05).mean(), 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- A p-value measures how surprising the data would be if the null were true.\n", "- Power depends on effect size relative to the SE; the MDE at 80% power is about 2.8 SEs.\n", "- Watch for multiple testing, tiny-but-significant effects and exaggerated significant estimates." ] } ] }