Day 7: The Law of Large Numbers & Central Limit Theorem

Theory

The sample mean

Let $X_1, \dots, X_n$ be independent and identically distributed (iid) with mean $\mu$ and variance $\sigma^2$ . The sample mean $\bar X_n = \frac1n\sum_i X_i$ has

E[\bar X_n] = \mu, \qquad \operatorname{Var}(\bar X_n) = \frac{\sigma^2}{n}, \qquad \operatorname{SE}(\bar X_n) = \frac{\sigma}{\sqrt n}.

The standard error shrinks like $1/\sqrt n$ : to halve it, you need four times the data.

Two inequalities

Markov: if $X \ge 0$ , then $P(X \ge a) \le E[X]/a$ for $a > 0$ .
Chebyshev: $P(|X - \mu| \ge k) \le \sigma^2/k^2$ .

Law of large numbers (LLN)

For every $\varepsilon > 0$ , $P(|\bar X_n - \mu| > \varepsilon) \to 0$ as $n \to \infty$ . We write $\bar X_n \xrightarrow{p} \mu$ ("converges in probability"). This is why a large experiment's group means are close to the population means.

Central limit theorem (CLT)

Whatever the shape of the distribution of $X$ (as long as $\sigma^2$ is finite),

\frac{\bar X_n - \mu}{\sigma/\sqrt n} \xrightarrow{d} N(0, 1).

For moderate $n$ , $\bar X_n \approx N(\mu, \sigma^2/n)$ . Almost every confidence interval and p-value in applied work rests on this approximation, including those for regression coefficients, which are weighted averages.

When things go wrong

Dependence. With $m$ units per cluster and intra-cluster correlation $\rho$ , the variance of the mean is inflated by the design effect $1 + (m - 1)\rho$ .
Infinite variance. For very fat-tailed data (e.g. Cauchy), neither the LLN nor the CLT applies.

Part 1 · Warm-up

Exercise 1 · Standard error

Test scores have $\sigma = 20$ . What is the standard error of a mean from $n = 100$ students?

Show solution

$\operatorname{SE} = 20/\sqrt{100} = 2$ .

Exercise 2 · Diminishing returns

How many students are needed to cut the standard error to 1?

Show solution

$20/\sqrt n = 1 \Rightarrow n = 400$ : four times as many for half the error.

Exercise 3 · A probability for the mean

$X$ has $\mu = 50$ and $\sigma = 10$ . For $n = 25$ , approximate $P(\bar X > 53)$ .

Show solution

$\operatorname{SE} = 10/5 = 2$ , $z = (53 - 50)/2 = 1.5$ , so $P \approx 1 - \Phi(1.5) \approx 0.067$ .

Exercise 4 · A proportion

The true employment rate is 0.3. For a sample of 400, find the standard error of the sample proportion and an interval containing it with about 95% probability.

Show solution

Step 1. A proportion is the mean of 0/1 variables with $\sigma^2 = p(1 - p) = 0.3 \times 0.7 = 0.21$ .

Step 2. Standard error: $\sqrt{0.21/400} = \sqrt{0.000525} \approx 0.0229$ .

Step 3. By the CLT, about 95% of samples fall within $1.96 \times 0.0229 \approx 0.045$ of 0.3.

Answer: SE ≈ 0.023; the sample employment rate lands in $[0.255,\; 0.345]$ about 95% of the time.

Exercise 5 · A total

An insurer has 100 independent claims with mean $500 and sd $300. Approximate the probability that total claims exceed $55,000.

Show solution

Step 1. Total $S = \sum_{i=1}^{100}X_i$ has mean $100 \times 500 = 50{,}000$ .

Step 2. Independent claims, so variances add: $\operatorname{Var}(S) = 100 \times 300^2$ and $\operatorname{sd}(S) = 300\sqrt{100} = 3{,}000$ .

Step 3. By the CLT, $S \approx N(50{,}000,\; 3{,}000^2)$ . Standardise: $z = (55{,}000 - 50{,}000)/3{,}000 \approx 1.67$ .

Step 4. $P(S > 55{,}000) \approx 1 - \Phi(1.67) \approx 0.048$ .

Answer: about a 5% chance that total claims exceed $55,000.

Part 2 · Proofs

Exercise 6 · Unbiasedness of the sample mean

Prove $E[\bar X_n] = \mu$ .

Show solution

By linearity, $E[\bar X_n] = \frac1n\sum_i E[X_i] = \frac1n(n\mu) = \mu$ . (Identical distribution is needed; independence is not.)

Exercise 7 · Variance of the sample mean

Prove $\operatorname{Var}(\bar X_n) = \sigma^2/n$ for iid data.

Show solution

\operatorname{Var}(\bar X_n) = \frac{1}{n^2}\operatorname{Var}\Big(\sum_i X_i\Big) = \frac{1}{n^2}\Big(\sum_i \operatorname{Var}(X_i) + \sum_{i \ne j}\operatorname{Cov}(X_i, X_j)\Big) = \frac{1}{n^2}\,n\sigma^2 = \frac{\sigma^2}{n},

because independence makes every covariance zero. Exercise 13 shows what happens when it does not.

Exercise 8 · Markov's inequality

Prove $P(X \ge a) \le E[X]/a$ for $X \ge 0$ and $a > 0$ .

Show solution

Step 1. Since $X \ge 0$ , $X \ge a\,\mathbf{1}\{X \ge a\}$ always holds.

Step 2. Take expectations: $E[X] \ge a\,P(X \ge a)$ .

Step 3. Divide by $a$ .

Exercise 9 · Chebyshev's inequality

Derive Chebyshev's inequality from Markov's.

Show solution

Apply Markov to the non-negative variable $(X - \mu)^2$ with $a = k^2$ :

P(|X - \mu| \ge k) = P\big((X - \mu)^2 \ge k^2\big) \le \frac{E[(X - \mu)^2]}{k^2} = \frac{\sigma^2}{k^2}.

Exercise 10 · The weak law of large numbers

Prove $\bar X_n \xrightarrow{p} \mu$ when $\sigma^2 < \infty$ .

Show solution

Apply Chebyshev to $\bar X_n$ , whose variance is $\sigma^2/n$ :

P(|\bar X_n - \mu| \ge \varepsilon) \le \frac{\sigma^2}{n\varepsilon^2} \to 0 \quad\text{as } n \to \infty.

Part 3 · Applied

Exercise 11 · Budgeting an RCT

An outcome has $\sigma = 30$ in both arms. (a) With 225 per arm, what is the standard error of the difference in means? (b) How many per arm are needed for a standard error of 2?

Show solution

(a) SE with 225 per arm

Step 1. Each arm mean has variance $\sigma^2/n = 900/225 = 4$ .

Step 2. The arms are independent, so the variance of the difference is $4 + 4 = 8$ and $\operatorname{SE} = \sqrt 8 \approx 2.83$ .

(b) Sample size for SE = 2

Step 3. Solve $\sqrt{2 \times 900/n} = 2$ : square both sides, $1800/n = 4$ , so $n = 450$ per arm.

Answer: SE ≈ 2.83 with 225 per arm; 450 per arm are needed for SE = 2 (twice the sample for a 29% smaller SE).

Exercise 12 · How sure is a survey mean?

A statistics office estimates mean household income from a random sample of $n$ households. Bound the probability that the sample mean misses the true mean by at least two standard errors, $P(|\bar X_n - \mu| \ge 2\sigma/\sqrt n)$ , using Chebyshev's inequality, and compare with the CLT approximation. Why might the office prefer the Chebyshev figure for very skewed incomes in a small sample?

Show solution

Step 1. Chebyshev with $\operatorname{Var}(\bar X_n) = \sigma^2/n$ and $k = 2\sigma/\sqrt n$ :

P(|\bar X_n - \mu| \ge k) \le \frac{\sigma^2/n}{4\sigma^2/n} = 0.25.

Step 2. CLT: $(\bar X_n - \mu)/(\sigma/\sqrt n) \approx N(0, 1)$ , so the probability is about $2(1 - \Phi(2)) \approx 0.046$ .

Answer: at most 25% (Chebyshev) versus about 4.6% (CLT). Chebyshev holds for any distribution and any $n$ , so it is a safe worst case when incomes are very skewed and the sample small; the CLT figure is sharper but only an approximation.

Exercise 13 · Clustered samples

A school survey samples 100 schools with 20 pupils each. The intra-class correlation is 0.05. Find the design effect and the effective sample size.

Show solution

Step 1. Design effect: $\text{DEFF} = 1 + (m - 1)\rho = 1 + 19 \times 0.05 = 1.95$ .

Step 2. Effective sample size: $2{,}000/1.95 \approx 1{,}026$ independent pupils.

Step 3. Standard errors that ignore clustering are too small by $\sqrt{1.95} \approx 1.40$ .

Answer: DEFF = 1.95; the 2,000 pupils carry the information of about 1,026 independent ones.

Exercise 14 · When the LLN fails: very fat tails

Suppose the profits of start-ups in a venture-capital portfolio follow a Cauchy distribution (no finite mean or variance). Explain why the average profit across more and more start-ups does not settle down, and what this suggests for heavy-tailed economic outcomes such as wealth or firm revenue.

Show solution

Step 1. The LLN and CLT both need a finite mean (and the CLT a finite variance). The Cauchy has neither.

Step 2. A known property: the average of $n$ independent Cauchy draws is itself Cauchy with the same spread, so averaging never reduces uncertainty.

Step 3. Intuition: every so often one start-up has a gigantic gain or loss that dominates the whole average (see the running means in the code).

Answer: the portfolio average does not converge. Real outcomes such as wealth have finite but huge variances, so convergence is slow and a few extreme units drive the mean; analysts use logs, winsorising, medians or quantiles.

Exercise 15 · Seeing the CLT in hospital stays

Hospital length of stay is strongly right-skewed: suppose it is exponential with mean 1 week. A health authority audits random samples of 50 patients and records each sample's mean stay. If you simulate 10,000 such audits, what mean, variance and shape should the sample means have?

Show solution

Step 1. For the exponential with mean 1, $\mu = 1$ and $\sigma^2 = 1$ .

Step 2. $E[\bar X] = \mu = 1$ week.

Step 3. $\operatorname{Var}(\bar X) = \sigma^2/n = 1/50 = 0.02$ , so sd ≈ 0.141 weeks.

Step 4. Shape: by the CLT, approximately normal. The skewness shrinks from 2 for one patient to about $2/\sqrt{50} \approx 0.28$ for the mean.

Answer: mean 1, variance 0.02, nearly bell-shaped despite the skewed stays (see the code and the CLT simulator).

Verify in Python

import numpy as np
from scipy import stats

rng = np.random.default_rng(2026)
means = rng.exponential(1.0, size=(10_000, 50)).mean(axis=1)
print(f"mean of means {means.mean():.3f} (theory 1), var {means.var():.4f} (theory 0.02)")
print(f"skewness of draws ~2, of means {stats.skew(means):.2f} (theory 2/sqrt(50) = 0.28)")

print("P(Xbar > 53):", round(stats.norm.sf(53, 50, 2), 4))
print("Total claims tail:", round(stats.norm.sf(55_000, 50_000, 3_000), 4))
print("Chebyshev bound 0.25 vs CLT:", round(2 * stats.norm.sf(2), 4))

# Cauchy: running mean never settles
c = rng.standard_cauchy(1_000_000)
print("Cauchy running means:", [round(float(c[:k].mean()), 2) for k in (10**3, 10**4, 10**5, 10**6)])

Key takeaways

Sample means are unbiased with standard error $\sigma/\sqrt n$ .
The LLN says they converge to $\mu$ ; the CLT says their errors are approximately normal.
Dependence (clusters) and fat tails weaken both results.

Day 7. The Law of Large Numbers & Central Limit Theorem

Theory

The sample mean

Two inequalities

Law of large numbers (LLN)

Central limit theorem (CLT)

When things go wrong

Part 1 · Warm-up

Exercise 1 · Standard error

Exercise 2 · Diminishing returns

Exercise 3 · A probability for the mean

Exercise 4 · A proportion

Exercise 5 · A total

Part 2 · Proofs

Exercise 6 · Unbiasedness of the sample mean

Exercise 7 · Variance of the sample mean

Exercise 8 · Markov's inequality

Exercise 9 · Chebyshev's inequality

Exercise 10 · The weak law of large numbers

Part 3 · Applied

Exercise 11 · Budgeting an RCT

Exercise 12 · How sure is a survey mean?

Exercise 13 · Clustered samples

Exercise 14 · When the LLN fails: very fat tails

Exercise 15 · Seeing the CLT in hospital stays

Verify in Python

Key takeaways

Simulation Lab: Central Limit Theorem Simulator