Day 10: Hypothesis Testing, p-Values & Power

Theory

The logic of a test

State a null hypothesis $H_0: \tau = 0$ and an alternative $H_1: \tau \ne 0$ .
Compute a test statistic whose distribution under $H_0$ is known: $z = \hat\tau/\operatorname{SE} \approx N(0, 1)$ .
The p-value is the probability, if $H_0$ were true, of a statistic at least as extreme as the one observed: $p = 2\big(1 - \Phi(|z|)\big)$ .
Reject $H_0$ at level $\alpha$ when $p < \alpha$ , equivalently $|z| > z_{1-\alpha/2}$ .

Two kinds of error

	$H_0$ true	$H_0$ false
Reject	Type I error (prob. $\alpha$ )	Correct (prob. = power)
Do not reject	Correct	Type II error (prob. $\beta$ )

Power, sample size and the MDE

If the true effect is $\tau$ , then $z \approx N(\tau/\operatorname{SE}, 1)$ and

\text{Power} \approx \Phi\Big(\frac{|\tau|}{\operatorname{SE}} - z_{1-\alpha/2}\Big).

For two arms of size $n$ with outcome SD $\sigma$ , $\operatorname{SE} = \sigma\sqrt{2/n}$ . Solving for power $1 - \beta$ :

n = \frac{2\sigma^2\,(z_{1-\alpha/2} + z_{1-\beta})^2}{\tau^2}, \qquad \text{MDE} = (z_{1-\alpha/2} + z_{1-\beta})\,\operatorname{SE}.

With $\alpha = 0.05$ and 80% power, $z_{0.975} + z_{0.8} = 1.96 + 0.84 = 2.80$ : the MDE is 2.8 standard errors.

Pitfalls

Multiple testing: with many tests, some will be "significant" by chance.
Statistical ≠ economic significance: huge samples make tiny effects significant.
Winner's curse: in underpowered studies, estimates that clear the significance bar are exaggerated.

Part 1 · Warm-up

Exercise 1 · Hypotheses

A job-training programme's effect on earnings is $\tau$ . Write the null and two-sided alternative for "the programme has an effect".

Show solution

$H_0: \tau = 0$ versus $H_1: \tau \ne 0$ .

Exercise 2 · A z-statistic

The estimated effect is 3.2 (thousand dollars) with SE 1.25. Compute $z$ .

Show solution

$z = 3.2/1.25 = 2.56$ .

Exercise 3 · Decision and p-value

Is the effect significant at 5%? Compute the p-value.

Show solution

$|z| = 2.56 > 1.96$ , so reject $H_0$ . $p = 2(1 - \Phi(2.56)) \approx 0.0105$ .

Exercise 4 · What a p-value is not

A report says "p = 0.01, so there is a 1% chance the programme has no effect." Correct it.

Show solution

The p-value is computed assuming $H_0$ is true: if the programme had no effect, results at least this extreme would occur about 1% of the time. It is not $P(H_0 \mid \text{data})$ ; that requires a prior (Day 13).

Exercise 5 · Trading off errors

What happens to Type I and Type II error rates if you lower $\alpha$ from 0.05 to 0.01 with the same data?

Show solution

Type I falls to 1%, but the critical value rises from 1.96 to 2.576, so real effects are rejected less often: Type II error rises and power falls. Only more information (larger $n$ , less noise) lowers both.

Part 2 · Proofs

Exercise 6 · p-values are uniform under the null

Let the statistic $Z$ have continuous CDF $F$ under $H_0$ , and define the one-sided p-value $p = 1 - F(Z)$ . Show $p \sim \text{Uniform}(0, 1)$ under $H_0$ .

Show solution

For $u \in (0, 1)$ : $P(p \le u) = P(F(Z) \ge 1 - u) = P(Z \ge F^{-1}(1 - u)) = 1 - F(F^{-1}(1 - u)) = u$ . So under $H_0$ every p-value is equally likely, and $P(p < 0.05) = 0.05$ by construction.

Exercise 7 · The power formula

Derive $\text{Power} = \Phi(\tau/\operatorname{SE} - z_{1-\alpha/2}) + \Phi(-\tau/\operatorname{SE} - z_{1-\alpha/2})$ .

Show solution

Step 1. Under the alternative, $z = \hat\tau/\operatorname{SE} \sim N(\delta, 1)$ with $\delta = \tau/\operatorname{SE}$ .

Step 2. With $c = z_{1-\alpha/2}$ :

P(z > c) = 1 - \Phi(c - \delta) = \Phi(\delta - c), \qquad P(z < -c) = \Phi(-c - \delta).

Step 3. Add them.

Step 4. For $\delta > 0$ the second term is tiny, giving the approximation in the theory block.

Exercise 8 · Sample size

Derive $n = 2\sigma^2(z_{1-\alpha/2} + z_{1-\beta})^2/\tau^2$ per arm, and apply it with $\sigma = 10$ , $\tau = 2$ , $\alpha = 0.05$ , power 0.8.

Show solution

Derivation

Step 1. Approximate power: $\Phi(\tau/\operatorname{SE} - z_{1-\alpha/2}) = 1 - \beta$ .

Step 2. Apply $\Phi^{-1}$ : $\tau/\operatorname{SE} = z_{1-\alpha/2} + z_{1-\beta}$ .

Step 3. Substitute $\operatorname{SE} = \sigma\sqrt{2/n}$ : $\tau\sqrt n/(\sigma\sqrt2) = z_{1-\alpha/2} + z_{1-\beta}$ .

Step 4. Square and solve: $n = 2\sigma^2(z_{1-\alpha/2} + z_{1-\beta})^2/\tau^2$ .

Application

Step 5. $z_{0.975} = 1.96$ , $z_{0.80} = 0.8416$ , so $(1.96 + 0.8416)^2 = 2.8016^2 \approx 7.849$ .

Step 6. $n = 2(100)(7.849)/4 \approx 392.4$ ; round up.

Answer: 393 per arm.

Exercise 9 · Unequal allocation

With a fixed total $N$ and a share $p$ treated, $\operatorname{Var}(\hat\tau) = \sigma^2/(Np(1 - p))$ . Show $p = 0.5$ is optimal and find the cost of a 70/30 split.

Show solution

Step 1. $\operatorname{Var}(\hat\tau) = \sigma^2/(Np(1 - p))$ is smallest when $p(1 - p)$ is largest, which happens at $p = 0.5$ (Day 3, Exercise 12).

Step 2. At a 70/30 split, $p(1 - p) = 0.21$ , versus 0.25 at 50/50.

Step 3. Variance ratio: $0.25/0.21 \approx 1.19$ .

Answer: 50/50 is optimal; a 70/30 split needs about 19% more people for the same precision (still worth it if treatment is expensive, Day 26).

Exercise 10 · The winner's curse

The true effect is $\tau = 1$ and $\operatorname{SE} = 1$ . Compute the power, and the expected estimate given that it is significantly positive. Hint: if $X \sim N(\mu, 1)$ , then $E[X \mid X > c] = \mu + \phi(c - \mu)/(1 - \Phi(c - \mu))$ .

Show solution

Step 1. Power: $\tau/\operatorname{SE} = 1$ , so power $\approx \Phi(1 - 1.96) = \Phi(-0.96) \approx 0.17$ .

Step 2. Given a significant positive result, use the truncated-normal mean with $\mu = 1$ , $c = 1.96$ , $c - \mu = 0.96$ :

E[\hat\tau \mid \hat\tau > 1.96] = 1 + \frac{\phi(0.96)}{1 - \Phi(0.96)}.

Step 3. $\phi(0.96) = 0.3989\,e^{-0.96^2/2} \approx 0.2516$ and $1 - \Phi(0.96) \approx 0.1685$ ; ratio ≈ 1.493.

Step 4. $E[\hat\tau \mid \text{significant}] \approx 2.49$ .

Answer: power is only 17%, and the significant estimates average 2.5 times the true effect. Small published studies often overstate effects for this reason.

Part 3 · Applied

Exercise 11 · One-sided or two-sided? A cash-transfer test

A conditional cash transfer's effect on school enrolment gives $z = 1.80$ . The team had planned a two-sided test. Compute the one-sided and two-sided p-values. Which should be reported?

Show solution

Step 1. One-sided: $P(Z > 1.80) = 1 - \Phi(1.80) \approx 0.036$ .

Step 2. Two-sided: $2 \times 0.036 = 0.072$ .

Answer: report the pre-planned two-sided $p = 0.072$ . Switching to one-sided after seeing the sign doubles the false-positive rate; one-sided tests are fine only if pre-registered for a substantive reason (e.g. the transfer cannot plausibly reduce enrolment).

Exercise 12 · An underpowered nutrition trial

A nutrition supplement trial can estimate the effect on child height with SE 1.5 cm. If the true effect is 3 cm, what is the trial's power at the 5% level?

Show solution

Step 1. Standardised effect: $\tau/\operatorname{SE} = 3/1.5 = 2$ .

Step 2. Power $\approx \Phi(2 - 1.96) = \Phi(0.04) \approx 0.52$ .

Answer: about 52%: a coin flip. A null result from this trial would say little about whether the supplement works.

Exercise 13 · Minimum detectable effect

With 500 per arm and $\sigma = 20$ , what is the MDE at 5% significance and 80% power?

Show solution

Step 1. $\operatorname{SE} = \sigma\sqrt{2/n} = 20\sqrt{2/500} = 20 \times 0.0632 \approx 1.265$ .

Step 2. MDE multiplier at 5% and 80% power: $1.96 + 0.84 = 2.80$ .

Step 3. $\text{MDE} = 2.80 \times 1.265 \approx 3.54$ .

Answer: the study can reliably detect effects of about 3.5 units or more.

Exercise 14 · Twenty outcomes

A study tests 20 independent outcomes at 5%, and the programme truly affects none. What is the chance of at least one "significant" result? What threshold does Bonferroni use?

Show solution

Step 1. Each test has a 0.95 chance of not giving a false positive; independent, so all 20 avoid one with probability $0.95^{20} \approx 0.358$ .

Step 2. At least one false positive: $1 - 0.358 \approx 0.64$ .

Step 3. Bonferroni divides the level by the number of tests: $0.05/20 = 0.0025$ . By Boole's inequality (Day 1), the chance of any false positive is then at most $20 \times 0.0025 = 0.05$ .

Answer: a 64% chance of at least one spurious "effect"; Bonferroni tests each outcome at 0.0025.

Exercise 15 · Statistical versus economic significance

On 5 million tax records, a policy raises earnings by $1.50 a year with $p < 0.0001$ . Interpret.

Show solution

Step 1. $z = \hat\tau/\operatorname{SE}$ with $\operatorname{SE} = \sigma/\sqrt n$ .

Step 2. With $n = 5{,}000{,}000$ , $\sqrt n \approx 2{,}236$ , so the SE is tiny and even $1.50 gives a huge $z$ .

Answer: the effect is precisely estimated (statistically significant) but negligible in size (economically insignificant). Report the effect and its CI, not just the p-value.

Verify in Python

import numpy as np
from scipy import stats

z = 3.2 / 1.25
print(f"z = {z:.2f}, p = {2 * stats.norm.sf(z):.4f}")

za, zb = stats.norm.ppf(0.975), stats.norm.ppf(0.80)
print("n per arm:", np.ceil(2 * 10**2 * (za + zb)**2 / 2**2))
print("MDE:", round((za + zb) * 20 * np.sqrt(2 / 500), 2))
print("P(>=1 false positive in 20):", round(1 - 0.95**20, 3))

# Winner's curse by simulation
rng = np.random.default_rng(10)
est = rng.normal(1, 1, 1_000_000)
sig = est > za
print(f"power {sig.mean():.3f}; mean significant estimate {est[sig].mean():.2f} (truth 1)")

# p-values are uniform under H0
p = 2 * stats.norm.sf(np.abs(rng.normal(0, 1, 100_000)))
print("share p < 0.05 under H0:", round((p < 0.05).mean(), 3))

Key takeaways

A p-value measures how surprising the data would be if the null were true.
Power depends on effect size relative to the SE; the MDE at 80% power is about 2.8 SEs.
Watch for multiple testing, tiny-but-significant effects and exaggerated significant estimates.

Day 10. Hypothesis Testing, p-Values & Power

Theory

The logic of a test

Two kinds of error

Power, sample size and the MDE

Pitfalls

Part 1 · Warm-up

Exercise 1 · Hypotheses

Exercise 2 · A z-statistic

Exercise 3 · Decision and p-value

Exercise 4 · What a p-value is not

Exercise 5 · Trading off errors

Part 2 · Proofs

Exercise 6 · p-values are uniform under the null

Exercise 7 · The power formula

Exercise 8 · Sample size

Exercise 9 · Unequal allocation

Exercise 10 · The winner's curse

Part 3 · Applied

Exercise 11 · One-sided or two-sided? A cash-transfer test

Exercise 12 · An underpowered nutrition trial

Exercise 13 · Minimum detectable effect

Exercise 14 · Twenty outcomes

Exercise 15 · Statistical versus economic significance

Verify in Python

Key takeaways

Simulation Lab: Statistical Power Visualiser