Math for Causal Inference
Phase 1 · Probability & Statistical Inference / Day 8

Day 8. Estimators: Bias, Variance, MSE & Consistency

How to judge an estimator, and why "unbiased" is not the same as "good".

Video coming soon
Learning outcomes
  • ●Compute the bias, variance and mean squared error (MSE) of an estimator.
  • ●Prove the bias–variance decomposition and that the n − 1 sample variance is unbiased.
  • ●Distinguish unbiasedness from consistency and show when MSE → 0 implies consistency.
  • ●Combine independent estimates with inverse-variance weights.
2-HOUR SESSION PLAN
Theory 25′Warm-up (1–5) 20′Proofs (6–10) 35′Applied (11–15) 25′Python & Lab 15′
Colab notebook (.ipynb) Jump to the simulator

Theory

Estimands, estimators, estimates

  • The estimand θ\theta is the quantity you want (the ATE, a population mean).
  • An estimator θ^\hat\theta is a rule that turns data into a number. Because data are random, θ^\hat\theta is a random variable with a sampling distribution.
  • An estimate is the number you get from your particular sample.

Three criteria

Bias⁡(θ^)=E[θ^]−θ,Var⁡(θ^)=E[(θ^−E[θ^])2],MSE⁡(θ^)=E[(θ^−θ)2]=Var⁡(θ^)+Bias⁡2.\operatorname{Bias}(\hat\theta) = E[\hat\theta] - \theta, \qquad \operatorname{Var}(\hat\theta) = E\big[(\hat\theta - E[\hat\theta])^2\big], \qquad \operatorname{MSE}(\hat\theta) = E[(\hat\theta - \theta)^2] = \operatorname{Var}(\hat\theta) + \operatorname{Bias}^2.

A slightly biased estimator with much lower variance can have lower MSE: this bias–variance trade-off is behind shrinkage, ridge regression and machine-learning nuisance models (Phase 6).

Consistency

θ^n\hat\theta_n is consistent if θ^n→pθ\hat\theta_n \xrightarrow{p} \theta. A sufficient condition is MSE⁡(θ^n)→0\operatorname{MSE}(\hat\theta_n) \to 0. Consistent estimators can be biased in finite samples (e.g. Xˉ+1/n\bar X + 1/n); unbiased estimators can be inconsistent (e.g. using only the first observation).

Most estimators in causal inference are n\sqrt n-consistent and asymptotically normal:

n(θ^n−θ)→dN(0,V),\sqrt n(\hat\theta_n - \theta) \xrightarrow{d} N(0, V),

so their standard error is approximately V/n\sqrt{V/n}.

The sample variance

The plug-in variance σ^2=1n∑(Xi−Xˉ)2\hat\sigma^2 = \frac1n\sum(X_i - \bar X)^2 is biased downward; the sample variance s2=1n−1∑(Xi−Xˉ)2s^2 = \frac{1}{n-1}\sum(X_i - \bar X)^2 is unbiased. The "lost" degree of freedom is the one used to estimate Xˉ\bar X.

Causal link. A biased estimator does not improve with more data: a confounded comparison converges to the wrong number. Identification (is the estimand recoverable at all?) comes before estimation (how precisely?).

Part 1 · Warm-up

Exercise 1 · Two unbiased estimators

From iid data with mean μ\mu and variance σ2\sigma^2, compare θ^1=Xˉ\hat\theta_1 = \bar X and θ^2=(X1+Xn)/2\hat\theta_2 = (X_1 + X_n)/2.

Show solution

Step 1. Both are unbiased: E[θ^2]=(μ+μ)/2=μE[\hat\theta_2] = (\mu + \mu)/2 = \mu.

Step 2. But Var⁡(θ^1)=σ2/n\operatorname{Var}(\hat\theta_1) = \sigma^2/n while Var⁡(θ^2)=σ2/2\operatorname{Var}(\hat\theta_2) = \sigma^2/2.

Step 3. For n>2n > 2, Xˉ\bar X is more precise; θ^2\hat\theta_2 throws data away.

Exercise 2 · MSE arithmetic

An estimator has bias 2 and variance 5. What is its MSE and root-MSE?

Show solution

MSE⁡=5+22=9\operatorname{MSE} = 5 + 2^2 = 9, root-MSE =3= 3.

Exercise 3 · A shrinkage estimator

Let θ^c=c Xˉ\hat\theta_c = c\,\bar X with σ2=100\sigma^2 = 100, n=25n = 25 and c=0.9c = 0.9. Compute its MSE when μ=10\mu = 10 and when μ=2\mu = 2, and compare with Xˉ\bar X.

Show solution

Step 1. Bias of cXˉc\bar X: E[cXˉ]−μ=(c−1)μE[c\bar X] - \mu = (c - 1)\mu. Variance: c2σ2/n=0.81×100/25=3.24c^2\sigma^2/n = 0.81 \times 100/25 = 3.24.

Step 2. For Xˉ\bar X itself: bias 0, variance 100/25=4100/25 = 4, so MSE =4= 4.

Step 3. If μ=10\mu = 10: bias =−0.1×10=−1= -0.1 \times 10 = -1, MSE =3.24+1=4.24>4= 3.24 + 1 = 4.24 > 4.

Step 4. If μ=2\mu = 2: bias =−0.2= -0.2, MSE =3.24+0.04=3.28<4= 3.24 + 0.04 = 3.28 < 4.

Answer: shrinking toward zero helps when the truth is near zero (MSE 3.28 versus 4) and hurts when it is far away (4.24 versus 4).

Exercise 4 · The plug-in variance

For n=10n = 10 and σ2=50\sigma^2 = 50, what is E[σ^2]E[\hat\sigma^2] for the plug-in (divide-by-nn) variance?

Show solution

E[σ^2]=n−1nσ2=0.9×50=45E[\hat\sigma^2] = \frac{n - 1}{n}\sigma^2 = 0.9 \times 50 = 45: a bias of −5-5 (Exercise 7 proves this).

Exercise 5 · Biased but consistent

Is θ~n=Xˉn+1/n\tilde\theta_n = \bar X_n + 1/n unbiased? Consistent?

Show solution

Biased: E[θ~n]=μ+1/nE[\tilde\theta_n] = \mu + 1/n. Consistent: bias →0\to 0 and variance σ2/n→0\sigma^2/n \to 0, so the MSE →0\to 0.

Part 2 · Proofs

Exercise 6 · Bias–variance decomposition

Prove MSE⁡(θ^)=Var⁡(θ^)+Bias⁡(θ^)2\operatorname{MSE}(\hat\theta) = \operatorname{Var}(\hat\theta) + \operatorname{Bias}(\hat\theta)^2.

Show solution

Add and subtract E[θ^]E[\hat\theta]:

E[(θ^−θ)2]=E[(θ^−Eθ^)2]+2(Eθ^−θ)E[θ^−Eθ^]⏟0+(Eθ^−θ)2.E[(\hat\theta - \theta)^2] = E[(\hat\theta - E\hat\theta)^2] + 2(E\hat\theta - \theta)\underbrace{E[\hat\theta - E\hat\theta]}_{0} + (E\hat\theta - \theta)^2.

This is the same algebra as Day 3, Exercise 10.

Exercise 7 · Why divide by n − 1

Prove E[∑i=1n(Xi−Xˉ)2]=(n−1)σ2E\big[\sum_{i=1}^n(X_i - \bar X)^2\big] = (n - 1)\sigma^2 for iid data.

Show solution

Use ∑(Xi−Xˉ)2=∑(Xi−μ)2−n(Xˉ−μ)2\sum(X_i - \bar X)^2 = \sum(X_i - \mu)^2 - n(\bar X - \mu)^2 (expand Xi−μ=(Xi−Xˉ)+(Xˉ−μ)X_i - \mu = (X_i - \bar X) + (\bar X - \mu); the cross term sums to zero). Taking expectations:

E[∑(Xi−Xˉ)2]=nσ2−n⋅σ2n=(n−1)σ2.E\Big[\sum(X_i - \bar X)^2\Big] = n\sigma^2 - n\cdot\frac{\sigma^2}{n} = (n - 1)\sigma^2.

Exercise 8 · MSE → 0 implies consistency

Prove that if MSE⁡(θ^n)→0\operatorname{MSE}(\hat\theta_n) \to 0 then θ^n→pθ\hat\theta_n \xrightarrow{p} \theta.

Show solution

Apply Markov's inequality to (θ^n−θ)2(\hat\theta_n - \theta)^2:

P(∣θ^n−θ∣≥ε)≤E[(θ^n−θ)2]ε2=MSE⁡(θ^n)ε2→0.P(|\hat\theta_n - \theta| \ge \varepsilon) \le \frac{E[(\hat\theta_n - \theta)^2]}{\varepsilon^2} = \frac{\operatorname{MSE}(\hat\theta_n)}{\varepsilon^2} \to 0.

Exercise 9 · The sample mean is the best linear unbiased estimator

Among estimators ∑iwiXi\sum_i w_iX_i with ∑iwi=1\sum_i w_i = 1 (so they are unbiased), show that equal weights minimise the variance.

Show solution

Step 1. The variance is σ2∑wi2\sigma^2\sum w_i^2.

Step 2. Write wi=1n+diw_i = \frac1n + d_i with ∑di=0\sum d_i = 0.

Step 3. Then

∑wi2=1n+2n∑di+∑di2=1n+∑di2≥1n,\sum w_i^2 = \frac1n + \frac2n\sum d_i + \sum d_i^2 = \frac1n + \sum d_i^2 \ge \frac1n,

Step 4. with equality only when all di=0d_i = 0.

Exercise 10 · The best shrinkage factor

For θ^c=cXˉ\hat\theta_c = c\bar X, find the cc that minimises the MSE.

Show solution

Step 1. MSE⁡(c)=c2σ2/n+(c−1)2μ2\operatorname{MSE}(c) = c^2\sigma^2/n + (c - 1)^2\mu^2.

Step 2. Setting the derivative to zero, 2cσ2/n+2(c−1)μ2=02c\sigma^2/n + 2(c - 1)\mu^2 = 0, gives

c∗=μ2μ2+σ2/n.c^* = \frac{\mu^2}{\mu^2 + \sigma^2/n}.

Step 3. It depends on the unknown μ\mu, which is why practical shrinkage (empirical Bayes, Day 13) estimates it from the data.

Part 3 · Applied

Exercise 11 · The difference in means is unbiased in a cash-transfer RCT

Households are randomly assigned to receive a cash transfer (T=1T = 1) or not, independently of their potential consumption levels (Y(1),Y(0))(Y(1), Y(0)). Show that the difference between the treated and control groups' mean consumption, Yˉ1−Yˉ0\bar Y_1 - \bar Y_0, is unbiased for the ATE.

Show solution

Step 1. In the treated group each observed Yi=Yi(1)Y_i = Y_i(1), so E[Yˉ1]=E[Y(1)∣T=1]E[\bar Y_1] = E[Y(1) \mid T = 1].

Step 2. Random assignment gives E[Y(1)∣T=1]=E[Y(1)]E[Y(1) \mid T = 1] = E[Y(1)] (Day 6, Exercise 14).

Step 3. Likewise E[Yˉ0]=E[Y(0)∣T=0]=E[Y(0)]E[\bar Y_0] = E[Y(0) \mid T = 0] = E[Y(0)].

Step 4. Linearity: E[Yˉ1−Yˉ0]=E[Y(1)]−E[Y(0)]=ATEE[\bar Y_1 - \bar Y_0] = E[Y(1)] - E[Y(0)] = \text{ATE}. ■\blacksquare

Exercise 12 · Pooling two evaluations

Two independent RCTs of the same school-meals programme estimate its effect on attendance as 2.0 days (SE 0.5) and 3.0 days (SE 1.0). Find the inverse-variance weighted estimate and its standard error.

Show solution

Step 1. Weights are inverse variances: w1=1/0.52=4w_1 = 1/0.5^2 = 4 and w2=1/1.02=1w_2 = 1/1.0^2 = 1.

Step 2. Pooled estimate:

θ^=4(2.0)+1(3.0)4+1=115=2.2.\hat\theta = \frac{4(2.0) + 1(3.0)}{4 + 1} = \frac{11}{5} = 2.2.

Step 3. Variance of the pooled estimate: 1/(w1+w2)=1/51/(w_1 + w_2) = 1/5, so SE⁡=1/5≈0.447\operatorname{SE} = 1/\sqrt5 \approx 0.447.

Answer: 2.2 days (SE 0.45), more precise than either study alone. This is fixed-effect meta-analysis.

Exercise 13 · More data does not cure confounding

In Day 4's confounded DGP the naive difference converges to 17 while the true effect is 5. Is the naive estimator consistent for the ATE?

Show solution

Step 1. The naive difference converges to its own probability limit, E[Y∣T=1]−E[Y∣T=0]=17E[Y \mid T = 1] - E[Y \mid T = 0] = 17.

Step 2. Its bias for the ATE is 17−5=1217 - 5 = 12 at every sample size.

Step 3. Its variance shrinks to zero, so its MSE tends to 0+122=1440 + 12^2 = 144, not zero.

Answer: not consistent for the ATE. More data gives an ever more precise estimate of the wrong number.

Exercise 14 · Mean or median test score?

A ministry summarises the test scores of 100 pupils per school. Scores are roughly normal with σ=10\sigma = 10. The sample median has variance about πσ2/(2n)\pi\sigma^2/(2n). Compare its precision with the sample mean's, and say when the median might be preferred.

Show solution

Step 1. Variance of the mean: σ2/n=100/100=1\sigma^2/n = 100/100 = 1.

Step 2. Variance of the median: πσ2/(2n)=π/2≈1.571\pi\sigma^2/(2n) = \pi/2 \approx 1.571.

Step 3. Relative efficiency: 1/1.571=2/π≈0.641/1.571 = 2/\pi \approx 0.64; the median needs about 57% more pupils for the same precision.

Answer: with normal scores the mean is more precise. With heavy tails or data errors (e.g. a few mis-coded scores of 0) the median can win; try the t-distribution in the Lab.

Exercise 15 · Planning a consumption survey

An unbiased estimator of mean monthly household consumption (in $100) has variance 4/n4/n. How many households are needed for its MSE to be below 0.01, i.e. a root-MSE below $10?

Show solution

Step 1. Unbiased, so MSE = variance = 4/n4/n.

Step 2. Require 4/n<0.014/n < 0.01, i.e. n>4/0.01=400n > 4/0.01 = 400.

Answer: more than 400 households.

Verify in Python

import numpy as np

rng = np.random.default_rng(8)
reps, n, sigma = 100_000, 25, 10
for mu in (10, 2):
    xbar = rng.normal(mu, sigma, (reps, n)).mean(1)
    for name, est in (("mean", xbar), ("0.9*mean", 0.9 * xbar)):
        print(f"mu={mu:>2} {name:>8}: bias {est.mean() - mu:+.3f}  MSE {((est - mu)**2).mean():.3f}")

# The n vs n-1 variance
x = rng.normal(0, np.sqrt(50), (100_000, 10))
print("E[plug-in var] ~", x.var(axis=1, ddof=0).mean().round(2), "; E[s^2] ~", x.var(axis=1, ddof=1).mean().round(2))

# Mean vs median efficiency with normal data
x = rng.normal(0, 10, (50_000, 100))
print("Var(mean) ~", x.mean(1).var().round(3), " Var(median) ~", np.median(x, 1).var().round(3))

Key takeaways

  • MSE = variance + bias²: judge estimators on both.
  • Consistency is about the limit, unbiasedness about the average: neither implies the other.
  • No sample size fixes a biased (unidentified) comparison.

Simulation Lab: Estimator Race: Bias, Variance & MSE

Open full screen →

Mean, median and a shrinkage estimator compete across repeated samples.

00.10.2-5-2.502.557.510estimatedensitytrue μ
Sample meanSample median0.90 × mean
estimatorbiasvarianceMSE
Sample mean0.0483.8473.849
Sample median0.0506.0276.029
0.90 × mean-0.1563.1163.141
Sample mean
Sample median
0.90 × mean
Solid = variance, light = bias². Lowest MSE: 0.90 × mean.

3,000 repeated samples per setting. With normal data the mean wins among unbiased estimators; with heavy tails (t₃) the median can beat it. Shrinkage wins when the truth is near zero.

Day 7Day 9