Day 36: Neyman's Repeated-Sampling Variance · Math for Causal Inference

Theory

The finite-population variance

Neyman (1923) treated potential outcomes as fixed and asked how the difference in means $\hat\tau = \bar Y_1 - \bar Y_0$ varies across all possible completely randomised assignments. With finite-population variances (divisor $n - 1$ )

S_1^2 = \tfrac{1}{n-1}\sum_i(Y_i(1) - \bar Y(1))^2, \quad S_0^2 = \tfrac{1}{n-1}\sum_i(Y_i(0) - \bar Y(0))^2, \quad S_\tau^2 = \tfrac{1}{n-1}\sum_i(\tau_i - \tau_{fp})^2,

the exact variance is

\operatorname{Var}(\hat\tau) = \frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{S_\tau^2}{n}.

Estimating it

$S_1^2$ and $S_0^2$ are estimated without bias by the sample variances $s_1^2$ (treated) and $s_0^2$ (control). But $S_\tau^2$ involves both potential outcomes of each unit and cannot be estimated. Neyman's estimator drops it:

\hat V = \frac{s_1^2}{n_1} + \frac{s_0^2}{n_0}, \qquad E[\hat V] - \operatorname{Var}(\hat\tau) = \frac{S_\tau^2}{n} \ge 0.

So $\hat V$ is conservative: unbiased when effects are constant ( $S_\tau^2 = 0$ ), too large otherwise. Confidence intervals $\hat\tau \pm 1.96\sqrt{\hat V}$ therefore cover at least 95% in large samples.

A sharper (still valid) bound uses $S_\tau^2 \ge (S_1 - S_0)^2$ .

Finite versus super population

If units are a random sample from a large population, the variance of $\hat\tau$ around the super-population ATE is $\sigma_1^2/n_1 + \sigma_0^2/n_0$ , with no negative term, and $\hat V$ is unbiased for it (Imbens & Rubin 2015). The same formula therefore serves both views: exact for one, conservative for the other.

Part 1 · Warm-up

Exercise 1 · Neyman's estimator

A cash-transfer RCT has 100 treated and 100 control households, with sample sds of monthly consumption $40 and $30. Compute $\hat V$ and the SE.

Show solution

Step 1. $\hat V = 40^2/100 + 30^2/100 = 16 + 9 = 25$ .

Step 2. SE $= \sqrt{25} = 5$ .

Answer: $5.

Exercise 2 · Constant effects

$S_1^2 = S_0^2 = 100$ , $n_1 = n_0 = 50$ , and the effect is the same for everyone. Compute $\operatorname{Var}(\hat\tau)$ .

Show solution

Step 1. Constant effects mean $S_\tau^2 = 0$ .

Step 2. $\operatorname{Var} = 100/50 + 100/50 - 0 = 4$ .

Answer: 4 (SE 2).

Exercise 3 · Heterogeneous effects

Same as Exercise 2 but $S_\tau^2 = 36$ ( $n = 100$ ). Compute the true variance and the bias of $\hat V$ .

Show solution

Step 1. $\operatorname{Var} = 2 + 2 - 36/100 = 3.64$ .

Step 2. $E[\hat V] = 4$ , so the bias is $0.36$ (about 10% of the variance; 5% of the SE).

Answer: 3.64; Neyman's estimator overstates it by 0.36.

Exercise 4 · A confidence interval

The estimated effect in Exercise 1 is $12. Build a 95% CI.

Show solution

Step 1. $12 \pm 1.96 \times 5 = 12 \pm 9.8$ .

Answer: $[2.2,\; 21.8]$ dollars per month: positive, but imprecise.

Exercise 5 · Allocating a fixed sample

With 200 households in total and outcome sds 40 (treated) and 30 (control), how should they be split to minimise $\hat V$ ?

Show solution

Step 1. Minimise $S_1^2/n_1 + S_0^2/n_0$ with $n_1 + n_0 = 200$ : the Lagrange solution (Day 26) is $n_1/n_0 = S_1/S_0 = 40/30$ .

Step 2. $n_1 = 200 \times 40/70 \approx 114$ , $n_0 \approx 86$ .

Answer: about 114 treated and 86 control.

Part 2 · Proofs

Exercise 6 · Unbiased group means

Under complete randomisation, prove $E[\bar Y_1] = \bar Y(1)$ .

Show solution

Step 1. $\bar Y_1 = \frac{1}{n_1}\sum_iT_iY_i(1)$ .

Step 2. $E[T_i] = n_1/n$ (Day 35, Exercise 7).

Step 3. $E[\bar Y_1] = \frac{1}{n_1}\cdot\frac{n_1}{n}\sum_iY_i(1) = \bar Y(1)$ . $\blacksquare$

Exercise 7 · Variance of the treated mean

Prove $\operatorname{Var}(\bar Y_1) = S_1^2\big(\frac{1}{n_1} - \frac1n\big)$ .

Show solution

Write $a_i = Y_i(1)$ and use Day 35's moments: $\operatorname{Var}(T_i) = \frac{n_1n_0}{n^2}$ , $\operatorname{Cov}(T_i, T_j) = -\frac{n_1n_0}{n^2(n - 1)}$ .

Step 1. $\operatorname{Var}\Big(\sum_iT_ia_i\Big) = \frac{n_1n_0}{n^2}\Big[\sum_ia_i^2 - \frac{1}{n - 1}\sum_{i \ne j}a_ia_j\Big]$ .

Step 2. Use $\sum_{i \ne j}a_ia_j = (\sum_ia_i)^2 - \sum_ia_i^2$ . The bracket becomes

\frac{n\sum_ia_i^2 - (\sum_ia_i)^2}{n - 1} = \frac{n\sum_i(a_i - \bar a)^2}{n - 1} = nS_1^2.

Step 3. So $\operatorname{Var}(\sum_iT_ia_i) = \frac{n_1n_0}{n}S_1^2$ .

Step 4. Divide by $n_1^2$ : $\operatorname{Var}(\bar Y_1) = \frac{n_0}{nn_1}S_1^2 = S_1^2\Big(\frac{1}{n_1} - \frac1n\Big)$ . $\blacksquare$

The $-1/n$ is a finite-population correction: treating everyone ( $n_1 = n$ ) gives zero variance.

Exercise 8 · The full Neyman formula (sketch)

Using Exercise 7 and $\operatorname{Cov}(\bar Y_1, \bar Y_0) = -S_{10}/n$ , where $S_{10}$ is the finite-population covariance of $Y(1)$ and $Y(0)$ , derive $\operatorname{Var}(\hat\tau)$ .

Show solution

Step 1. $\operatorname{Var}(\hat\tau) = \operatorname{Var}(\bar Y_1) + \operatorname{Var}(\bar Y_0) - 2\operatorname{Cov}(\bar Y_1, \bar Y_0)$ .

Step 2. Substitute: $S_1^2\big(\frac{1}{n_1} - \frac1n\big) + S_0^2\big(\frac{1}{n_0} - \frac1n\big) + \frac{2S_{10}}{n}$ .

Step 3. Since $\tau_i = Y_i(1) - Y_i(0)$ , $S_\tau^2 = S_1^2 + S_0^2 - 2S_{10}$ .

Step 4. So $-\frac{S_1^2 + S_0^2 - 2S_{10}}{n} = -\frac{S_\tau^2}{n}$ , giving $\operatorname{Var}(\hat\tau) = \frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{S_\tau^2}{n}$ . $\blacksquare$

Exercise 9 · Neyman's estimator is conservative

Given $E[s_1^2] = S_1^2$ and $E[s_0^2] = S_0^2$ under complete randomisation, prove $E[\hat V] \ge \operatorname{Var}(\hat\tau)$ , with equality iff effects are constant.

Show solution

Step 1. $E[\hat V] = S_1^2/n_1 + S_0^2/n_0$ .

Step 2. Subtract the true variance: $E[\hat V] - \operatorname{Var}(\hat\tau) = S_\tau^2/n \ge 0$ .

Step 3. $S_\tau^2 = 0$ iff all $\tau_i$ equal $\tau_{fp}$ , i.e. constant effects. $\blacksquare$

Exercise 10 · A sharper bound

Prove $S_\tau^2 \ge (S_1 - S_0)^2$ and deduce a smaller conservative variance estimator.

Show solution

Step 1. $S_\tau^2 = S_1^2 + S_0^2 - 2S_{10}$ .

Step 2. Cauchy–Schwarz (Day 16): $S_{10} \le S_1S_0$ .

Step 3. So $S_\tau^2 \ge S_1^2 + S_0^2 - 2S_1S_0 = (S_1 - S_0)^2$ .

Step 4. Hence $\operatorname{Var}(\hat\tau) \le \frac{S_1^2}{n_1} + \frac{S_0^2}{n_0} - \frac{(S_1 - S_0)^2}{n}$ , estimated by plugging in $s_1, s_0$ . $\blacksquare$

The bound is attained when $Y(1)$ and $Y(0)$ are perfectly positively correlated.

Part 3 · Applied

Exercise 11 · School attendance

A deworming RCT randomises 50 schools, 25 to each arm. Attendance rates (%) have sds 8 (treated) and 6 (control); the difference in means is 5 points. Compute the Neyman SE and a 95% CI.

Show solution

Step 1. $\hat V = 8^2/25 + 6^2/25 = 2.56 + 1.44 = 4$ .

Step 2. SE $= 2$ .

Step 3. CI: $5 \pm 1.96 \times 2 = [1.08,\; 8.92]$ .

Answer: attendance rose by 5 points (95% CI 1.1 to 8.9).

Exercise 12 · Using the sharper bound

For Exercise 11, compute the variance bound of Exercise 10.

Show solution

Step 1. $(s_1 - s_0)^2/n = (8 - 6)^2/50 = 0.08$ .

Step 2. Bound $= 4 - 0.08 = 3.92$ , SE $\approx 1.98$ .

Answer: a slightly narrower but still valid interval; the gain is small unless the two sds differ a lot.

Exercise 13 · The exact variance for the training table

For Day 31's four workers ( $Y(1) = (14, 12, 13, 15)$ , $Y(0) = (10, 12, 8, 15)$ , $n_1 = n_0 = 2$ ), compute $S_1^2$ , $S_0^2$ , $S_\tau^2$ and Neyman's exact variance. Check it against the variance of the six estimates from Day 31, Exercise 11.

Show solution

Step 1. $\bar Y(1) = 13.5$ ; deviations $0.5, -1.5, -0.5, 1.5$ ; $S_1^2 = (0.25 + 2.25 + 0.25 + 2.25)/3 = 5/3 \approx 1.667$ .

Step 2. $\bar Y(0) = 11.25$ ; squared deviations $1.5625, 0.5625, 10.5625, 14.0625$ ; $S_0^2 = 26.75/3 \approx 8.917$ .

Step 3. $\tau = (4, 0, 5, 0)$ , mean 2.25; squared deviations $3.0625, 5.0625, 7.5625, 5.0625$ ; $S_\tau^2 = 20.75/3 \approx 6.917$ .

Step 4. $\operatorname{Var} = 1.667/2 + 8.917/2 - 6.917/4 = 0.833 + 4.458 - 1.729 = 3.5625$ .

Step 5. The six estimates (1.5, 0, 4.5, 0, 4.5, 3) have mean 2.25 and variance $\frac16\sum(\hat\tau - 2.25)^2 = 3.5625$ . ✓

Answer: 3.5625 exactly, as Neyman's formula says.

Exercise 14 · A binary outcome

A smoking-cessation trial has 200 per arm; 30% quit with the intervention and 40% with usual care. Compute the difference, Neyman SE and z-statistic.

Show solution

Step 1. Difference $= 0.30 - 0.40 = -0.10$ .

Step 2. Binary outcomes: $s^2 \approx p(1 - p)$ . $\hat V = 0.21/200 + 0.24/200 = 0.00225$ .

Step 3. SE $\approx 0.0474$ ; $z = -0.10/0.0474 \approx -2.11$ .

Answer: the intervention lowered quit rates by 10 points ( $p \approx 0.035$ ), a surprising result worth checking for implementation problems.

Exercise 15 · Which variance to report?

A study randomises all 120 municipalities of a province (finite population, Day 31 Exercise 14). Is $\hat V$ exact, conservative or anti-conservative, and how should results be described?

Show solution

Step 1. For the finite-population ATE, $\hat V$ overstates the variance by $S_\tau^2/n$ on average: conservative.

Step 2. If effects are roughly constant across municipalities, the overstatement is negligible.

Answer: report $\hat V$ -based CIs as (slightly) conservative for the effect in these 120 municipalities; for other provinces, external validity applies.

Verify in Python

import itertools
import numpy as np

y1 = np.array([14, 12, 13, 15.]); y0 = np.array([10, 12, 8, 15.]); n, n1 = 4, 2
S1, S0, St = y1.var(ddof=1), y0.var(ddof=1), (y1 - y0).var(ddof=1)
print(f"S1^2 {S1:.3f}, S0^2 {S0:.3f}, Stau^2 {St:.3f}; Neyman variance {S1 / n1 + S0 / (n - n1) - St / n:.4f}")
ests = []
for c in itertools.combinations(range(n), n1):
    T = np.zeros(n, bool); T[list(c)] = True
    ests.append(y1[T].mean() - y0[~T].mean())
print("variance over all assignments:", np.var(ests))

# Coverage of Neyman intervals with heterogeneous effects
rng = np.random.default_rng(36)
N, N1 = 200, 100
Y0 = rng.normal(50, 10, N); Y1 = Y0 + rng.normal(5, 6, N)   # heterogeneous effects
tau = (Y1 - Y0).mean(); cover = []; vhat = []
for _ in range(20_000):
    T = np.zeros(N, bool); T[rng.choice(N, N1, replace=False)] = True
    est = Y1[T].mean() - Y0[~T].mean()
    v = Y1[T].var(ddof=1) / N1 + Y0[~T].var(ddof=1) / (N - N1)
    vhat.append(v); cover.append(abs(est - tau) <= 1.96 * np.sqrt(v))
true_v = Y1.var(ddof=1) / N1 + Y0.var(ddof=1) / (N - N1) - (Y1 - Y0).var(ddof=1) / N
print(f"true variance {true_v:.3f}, mean Neyman estimate {np.mean(vhat):.3f}, coverage {np.mean(cover):.3f}")

Key takeaways

Over all randomisations, $\operatorname{Var}(\hat\tau) = S_1^2/n_1 + S_0^2/n_0 - S_\tau^2/n$ .
The last term needs both potential outcomes per unit, so it is dropped: Neyman's estimator is conservative.
It is exact under constant effects and unbiased for the super-population variance.

Day 36. Neyman's Repeated-Sampling Variance

Theory

The finite-population variance

Estimating it

Finite versus super population

Part 1 · Warm-up

Exercise 1 · Neyman's estimator

Exercise 2 · Constant effects

Exercise 3 · Heterogeneous effects

Exercise 4 · A confidence interval

Exercise 5 · Allocating a fixed sample

Part 2 · Proofs

Exercise 6 · Unbiased group means

Exercise 7 · Variance of the treated mean

Exercise 8 · The full Neyman formula (sketch)

Exercise 9 · Neyman's estimator is conservative

Exercise 10 · A sharper bound

Part 3 · Applied

Exercise 11 · School attendance

Exercise 12 · Using the sharper bound

Exercise 13 · The exact variance for the training table

Exercise 14 · A binary outcome

Exercise 15 · Which variance to report?

Verify in Python

Key takeaways

Simulation Lab: Neyman's Conservative Variance