Day 9: Confidence Intervals & the Bootstrap · Math for Causal Inference

Theory

The pivot

If $\hat\theta \approx N(\theta, \operatorname{SE}^2)$ , then $(\hat\theta - \theta)/\operatorname{SE} \approx N(0, 1)$ , whose distribution does not depend on $\theta$ (a pivot). So

P\big(-z_{1-\alpha/2} \le \tfrac{\hat\theta - \theta}{\operatorname{SE}} \le z_{1-\alpha/2}\big) \approx 1 - \alpha \;\Longrightarrow\; \text{CI}_{1-\alpha} = \hat\theta \pm z_{1-\alpha/2}\,\operatorname{SE}.

Common critical values: $z_{0.95} = 1.645$ (90%), $z_{0.975} = 1.96$ (95%), $z_{0.995} = 2.576$ (99%).

With small samples from a normal population and an estimated SD, replace $z$ by the Student $t$ quantile with $n - 1$ degrees of freedom.

Interpretation

The interval is random; $\theta$ is fixed. "95% confidence" means that the procedure captures $\theta$ in 95% of repeated samples. It does not mean there is a 95% probability that $\theta$ lies in the interval you computed (that is a Bayesian credible interval, Day 13). The CI coverage simulator in the Lab makes this concrete.

Standard errors you will use constantly

Estimator	Standard error
Mean	$s/\sqrt n$
Proportion	$\sqrt{\hat p(1 - \hat p)/n}$
Difference in means (independent groups)	$\sqrt{s_1^2/n_1 + s_0^2/n_0}$

The nonparametric bootstrap

When the SE formula is unknown or messy (medians, ratios, complex estimators):

Draw $n$ observations with replacement from your sample.
Compute the estimator on this resample, $\hat\theta^{*}_b$ .
Repeat $B$ times (say $B = 2{,}000$ ).
Use the standard deviation of the $\hat\theta^{*}_b$ as the SE, or their 2.5th and 97.5th percentiles as a percentile interval.

The bootstrap treats the sample as a stand-in for the population. Resample in the way the data were generated: with clustered data, resample whole clusters.

Part 1 · Warm-up

Exercise 1 · A 95% interval for a mean

$\bar X = 52$ , $s = 10$ , $n = 100$ . Build a 95% CI.

Show solution

$\operatorname{SE} = 1$ , so $52 \pm 1.96 = [50.04,\; 53.96]$ .

Exercise 2 · Raising the confidence level

Build the 99% CI for the same data. Why is it wider?

Show solution

$52 \pm 2.576 = [49.42,\; 54.58]$ . More confidence requires a wider net.

Exercise 3 · A proportion

In a survey of 600, 42% support a policy. Build a 95% CI.

Show solution

Step 1. $\operatorname{SE} = \sqrt{\hat p(1 - \hat p)/n} = \sqrt{0.42 \times 0.58/600} = \sqrt{0.000406} \approx 0.0201$ .

Step 2. Margin of error: $1.96 \times 0.0201 \approx 0.0395$ .

Step 3. Interval: $0.42 \pm 0.0395$ .

Answer: $[0.381,\; 0.459]$ : support is between about 38% and 46%.

Exercise 4 · A treatment effect

Treated: mean 78, sd 12, $n = 144$ . Control: mean 74, sd 15, $n = 225$ . Build a 95% CI for the difference.

Show solution

Step 1. Point estimate: $78 - 74 = 4$ .

Step 2. Standard error with independent groups:

\operatorname{SE} = \sqrt{\frac{12^2}{144} + \frac{15^2}{225}} = \sqrt{1 + 1} \approx 1.414.

Step 3. Margin: $1.96 \times 1.414 \approx 2.77$ .

Answer: $4 \pm 2.77 = [1.23,\; 6.77]$ ; zero is excluded, so the effect is significant at 5%.

Exercise 5 · A small sample

$n = 16$ , $\bar X = 10$ , $s = 4$ . Build a 95% t interval ( $t_{15, 0.975} = 2.131$ ).

Show solution

$\operatorname{SE} = 4/4 = 1$ , so $10 \pm 2.131 = [7.87,\; 12.13]$ , slightly wider than the normal interval $10 \pm 1.96$ .

Part 2 · Proofs

Exercise 6 · Coverage of the known-σ interval

If $X_i \sim N(\mu, \sigma^2)$ iid with known $\sigma$ , prove that $\bar X \pm 1.96\,\sigma/\sqrt n$ contains $\mu$ with probability exactly 0.95.

Show solution

Step 1. $Z = (\bar X - \mu)/(\sigma/\sqrt n) \sim N(0, 1)$ exactly.

Step 2. The event $\mu \in [\bar X - 1.96\,\sigma/\sqrt n,\; \bar X + 1.96\,\sigma/\sqrt n]$ is the same event as $|Z| \le 1.96$ , whose probability is $\Phi(1.96) - \Phi(-1.96) = 0.95$ .

Exercise 7 · Width and sample size

Show that the width of a CI is proportional to $1/\sqrt n$ , and find the sample size needed to halve it.

Show solution

Width $= 2z\,\sigma/\sqrt n$ . Replacing $n$ by $kn$ multiplies it by $1/\sqrt k$ ; halving needs $k = 4$ .

Exercise 8 · Left out of the bootstrap

Show that the probability a given observation does not appear in a bootstrap resample is $(1 - 1/n)^n$ , and find its limit.

Show solution

Step 1. Each of the $n$ independent draws misses the observation with probability $1 - 1/n$ , so it is never drawn with probability $(1 - 1/n)^n$ .

Step 2. Since $\ln(1 - 1/n)^n = n\ln(1 - 1/n) \to -1$ , the limit is $e^{-1} \approx 0.368$ .

Step 3. Each resample contains about 63.2% of the distinct observations.

Exercise 9 · Duality of tests and intervals

Show that a two-sided 5% test of $H_0: \theta = \theta_0$ rejects exactly when $\theta_0$ lies outside the 95% CI.

Show solution

Step 1. The test rejects when $|\hat\theta - \theta_0|/\operatorname{SE} > 1.96$ , i.e. when $\theta_0 < \hat\theta - 1.96\operatorname{SE}$ or $\theta_0 > \hat\theta + 1.96\operatorname{SE}$ , which is exactly "outside the CI".

Step 2. A CI is the set of null values you would not reject.

Exercise 10 · Reverse-engineering a paper

A paper reports an effect with 95% CI $[1.2,\; 5.6]$ . Recover the estimate, its SE and its z-statistic.

Show solution

Step 1. A 95% CI is symmetric around the estimate: $\hat\theta = (1.2 + 5.6)/2 = 3.4$ .

Step 2. Its half-width is $1.96 \times \operatorname{SE}$ : $\operatorname{SE} = (5.6 - 1.2)/(2 \times 1.96) = 4.4/3.92 \approx 1.12$ .

Step 3. $z = 3.4/1.12 \approx 3.03$ , so $p = 2(1 - \Phi(3.03)) \approx 0.002$ .

Answer: estimate 3.4, SE 1.12, $z \approx 3.0$ , $p \approx 0.002$ .

Part 3 · Applied

Exercise 11 · Clustered data

Using Day 7's design effect of 1.95, by how much should a naive CI be widened?

Show solution

Step 1. Clustering multiplies the variance by the design effect, 1.95, so the SE must be multiplied by $\sqrt{1.95} \approx 1.40$ .

Step 2. The naive interval uses $\pm1.96$ naive SEs, which is only $1.96/1.40 = 1.40$ correct SEs.

Step 3. Its true coverage is $2\Phi(1.40) - 1 \approx 0.84$ .

Answer: widen it by 40%; otherwise the "95%" interval covers only about 84% of the time.

Exercise 12 · A percentile interval for median rent

A housing agency bootstraps the median monthly rent in a city from a sample of 300 listings, computes 1,000 bootstrap medians and sorts them. Which sorted values form the 95% percentile interval, and how would you report it?

Show solution

Step 1. A 95% interval leaves 2.5% in each tail.

Step 2. 2.5% of 1,000 is 25 and 97.5% is 975, so take roughly the 25th and 975th sorted bootstrap medians (numpy.percentile interpolates between neighbours).

Answer: report "median rent $X (95% bootstrap CI: [25th value, 975th value])". The interval need not be symmetric around $X, which is useful for skewed rents.

Exercise 13 · Bootstrapping a median

Health costs are heavily right-skewed. Why bootstrap the median rather than use a formula, and what are the steps?

Show solution

Step 1. Why not a formula: the median's analytic SE depends on the density of costs at the median, which is hard to estimate with skewed data.

Step 2. Draw 300 patients with replacement from the sample and compute the median cost.

Step 3. Repeat 2,000 times.

Step 4. Report the SD of the 2,000 medians as the SE, or their 2.5th and 97.5th percentiles as a 95% interval.

Answer: the bootstrap needs no formula, only the ability to recompute the median.

Exercise 14 · Overlapping intervals for two regions

A labour-force survey estimates mean hourly wages of $10 (SE 1) in region A and $13 (SE 1) in region B, from independent samples. Their 95% CIs overlap. Is the difference significant?

Show solution

Step 1. The CIs: A $10 \pm 1.96 = [8.04, 11.96]$ ; B $13 \pm 1.96 = [11.04, 14.96]$ . They overlap on $[11.04, 11.96]$ .

Step 2. Test the difference directly: $13 - 10 = 3$ with $\operatorname{SE} = \sqrt{1^2 + 1^2} \approx 1.41$ .

Step 3. $z = 3/1.41 \approx 2.12$ , $p \approx 0.034$ .

Answer: significant at 5% despite the overlap. Overlapping intervals are not a test; always test the difference.

Exercise 15 · What the bootstrap SE of mean income estimates

You bootstrap the mean income of $n$ surveyed households. Show that, if you could average over all possible resamples, the bootstrap variance of the mean equals $\hat\sigma^2/n$ , where $\hat\sigma^2 = \frac1n\sum(X_i - \bar X)^2$ .

Show solution

Step 1. Each draw in a resample picks one of the $n$ households with equal probability $1/n$ .

Step 2. So one draw has mean $\frac1n\sum X_i = \bar X$ and variance $\frac1n\sum(X_i - \bar X)^2 = \hat\sigma^2$ .

Step 3. A resample mean averages $n$ independent such draws, so its variance is $\hat\sigma^2/n$ .

Answer: the bootstrap variance is $\hat\sigma^2/n$ , essentially the textbook $s^2/n$ (smaller by the factor $(n - 1)/n$ ).

Verify in Python

import numpy as np
from scipy import stats

rng = np.random.default_rng(9)

# Coverage of the 95% t interval with skewed data (log-normal, true mean known)
mu_true = np.exp(0.5)
reps, n = 20_000, 30
x = rng.lognormal(0, 1, (reps, n))
m, s = x.mean(1), x.std(1, ddof=1)
t = stats.t.ppf(0.975, n - 1)
cover = np.mean((m - t * s / np.sqrt(n) <= mu_true) & (mu_true <= m + t * s / np.sqrt(n)))
print(f"coverage with skewed data, n=30: {cover:.3f}  (nominal 0.95)")
# With strongly skewed data and n = 30 the CLT has not fully kicked in: coverage falls short.

# Bootstrap the median of one sample
sample = rng.lognormal(8, 1, 200)            # health costs
boots = np.array([np.median(rng.choice(sample, sample.size)) for _ in range(2000)])
print("median:", round(np.median(sample)), " bootstrap SE:", round(boots.std(ddof=1)),
      " 95% percentile CI:", np.percentile(boots, [2.5, 97.5]).round())

print("P(left out):", round((1 - 1/200)**200, 4), " vs e^-1 =", round(np.exp(-1), 4))

Key takeaways

$\hat\theta \pm 1.96\,\operatorname{SE}$ covers $\theta$ in about 95% of repeated samples.
The bootstrap estimates a sampling distribution by resampling your data the way they were collected.
Compare estimates by testing their difference, not by checking whether intervals overlap.

Day 9. Confidence Intervals & the Bootstrap

Theory

The pivot

Interpretation

Standard errors you will use constantly

The nonparametric bootstrap

Part 1 · Warm-up

Exercise 1 · A 95% interval for a mean

Exercise 2 · Raising the confidence level

Exercise 3 · A proportion

Exercise 4 · A treatment effect

Exercise 5 · A small sample

Part 2 · Proofs

Exercise 6 · Coverage of the known-σ interval

Exercise 7 · Width and sample size

Exercise 8 · Left out of the bootstrap

Exercise 9 · Duality of tests and intervals

Exercise 10 · Reverse-engineering a paper

Part 3 · Applied

Exercise 11 · Clustered data

Exercise 12 · A percentile interval for median rent

Exercise 13 · Bootstrapping a median

Exercise 14 · Overlapping intervals for two regions

Exercise 15 · What the bootstrap SE of mean income estimates

Verify in Python

Key takeaways

Simulation Lab: Confidence-Interval Coverage