Day 40: Cluster Randomisation & Design Effects

Theory

When to randomise clusters

Randomise groups (schools, clinics, villages, markets) when the intervention is delivered to groups (teacher training, a clinic protocol), when spillovers within groups are likely (Day 32), or when individual randomisation is politically or logistically impossible.

The intra-cluster correlation

Write outcomes as $Y_{ig} = \mu + u_g + e_{ig}$ , with cluster effects $u_g$ (variance $\sigma_u^2$ ) and individual noise $e_{ig}$ (variance $\sigma_e^2$ ). Total variance $\sigma^2 = \sigma_u^2 + \sigma_e^2$ and the intra-cluster correlation

\rho = \frac{\sigma_u^2}{\sigma_u^2 + \sigma_e^2}

is the correlation between two people in the same cluster. Typical values: 0.01–0.05 for many health outcomes, 0.1–0.3 for test scores within schools.

Variance and design effect

With clusters of size $m$ , each cluster mean has variance $\sigma^2[\rho + (1 - \rho)/m]$ . With $J_1$ treated and $J_0$ control clusters,

\operatorname{Var}(\hat\tau) = \sigma^2\Big[\rho + \frac{1 - \rho}{m}\Big]\Big(\frac{1}{J_1} + \frac{1}{J_0}\Big) = \underbrace{[1 + (m - 1)\rho]}_{\text{design effect}}\times\sigma^2\Big(\frac{1}{mJ_1} + \frac{1}{mJ_0}\Big).

The design effect (DEFF) inflates the individually randomised variance. As $m \to \infty$ the variance falls only to $\sigma^2\rho(1/J_1 + 1/J_0)$ : adding clusters helps far more than adding people per cluster.

Power

With a share $P$ of $J$ clusters treated,

\text{MDE} = (z_{1-\alpha/2} + z_{1-\beta})\sqrt{\frac{\sigma^2[\rho + (1 - \rho)/m]}{P(1 - P)J}}.

Baseline covariates measured at the cluster level reduce the between-cluster variance and are especially valuable.

Analysis

Use cluster-robust SEs (Day 29), or analyse cluster means; with few clusters, use randomisation inference at the cluster level (Day 37) or the wild cluster bootstrap. With unequal cluster sizes, decide whether the estimand weights individuals or clusters equally.

Part 1 · Warm-up

Exercise 1 · A design effect

A school-based trial has 30 pupils per school and an ICC of 0.10. Compute the design effect.

Show solution

Step 1. DEFF $= 1 + (m - 1)\rho = 1 + 29 \times 0.10 = 3.9$ .

Answer: the variance is 3.9 times that of an individually randomised trial of the same size.

Exercise 2 · Effective sample size

With 60 schools of 30 pupils (Exercise 1), what is the effective sample size?

Show solution

Step 1. Total pupils: $60 \times 30 = 1{,}800$ .

Step 2. Effective $n = 1{,}800/3.9 \approx 462$ .

Answer: 1,800 pupils carry the information of about 462 independent ones.

Exercise 3 · SE and MDE

Outcomes are standardised ( $\sigma = 1$ ), $\rho = 0.1$ , $m = 30$ , and 30 schools per arm. Compute the SE and the MDE at 5% significance and 80% power.

Show solution

Step 1. Cluster-mean variance factor: $\rho + (1 - \rho)/m = 0.1 + 0.9/30 = 0.13$ .

Step 2. $\operatorname{Var}(\hat\tau) = 0.13 \times (1/30 + 1/30) = 0.13 \times 0.0667 \approx 0.00867$ ; SE $\approx 0.093$ .

Step 3. MDE $= 2.80 \times 0.093 \approx 0.26$ sd.

Answer: SE ≈ 0.093 sd; MDE ≈ 0.26 sd.

Exercise 4 · More pupils or more schools?

Starting from Exercise 3, compare (a) doubling pupils per school to 60 and (b) doubling schools to 60 per arm.

Show solution

Step 1. (a) Factor $0.1 + 0.9/60 = 0.115$ ; variance $0.115 \times 0.0667 \approx 0.00767$ ; SE $\approx 0.088$ ; MDE $\approx 0.245$ .

Step 2. (b) Factor 0.13; variance $0.13 \times (2/60) \approx 0.00433$ ; SE $\approx 0.066$ ; MDE $\approx 0.18$ .

Answer: both double the sample, but more schools cuts the MDE by 29% versus 6% for more pupils per school.

Exercise 5 · Estimating the ICC

Between-clinic variance of patient satisfaction is 0.2 and within-clinic variance 0.8. Compute the ICC.

Show solution

Step 1. $\rho = 0.2/(0.2 + 0.8) = 0.2$ .

Answer: 0.2: two patients of the same clinic have correlated satisfaction scores.

Part 2 · Proofs

Exercise 6 · Variance of a cluster mean

Under $Y_{ig} = \mu + u_g + e_{ig}$ with independent components, prove $\operatorname{Var}(\bar Y_g) = \sigma^2[\rho + (1 - \rho)/m]$ .

Show solution

Step 1. $\bar Y_g = \mu + u_g + \bar e_g$ , where $\bar e_g$ averages $m$ independent errors.

Step 2. $\operatorname{Var}(\bar Y_g) = \sigma_u^2 + \sigma_e^2/m$ .

Step 3. Substitute $\sigma_u^2 = \rho\sigma^2$ and $\sigma_e^2 = (1 - \rho)\sigma^2$ : $\sigma^2[\rho + (1 - \rho)/m]$ . $\blacksquare$

Exercise 7 · The design effect

Show that $\sigma^2[\rho + (1 - \rho)/m] = [1 + (m - 1)\rho]\,\sigma^2/m$ .

Show solution

Step 1. Factor out $\sigma^2/m$ : $\sigma^2[\rho + (1 - \rho)/m] = \frac{\sigma^2}{m}[m\rho + 1 - \rho]$ .

Step 2. $m\rho + 1 - \rho = 1 + (m - 1)\rho$ . $\blacksquare$

The second form is "individual-level variance $\sigma^2/m$ " times the DEFF.

Exercise 8 · The ceiling on cluster size

Prove that as $m \to \infty$ with $J_1, J_0$ fixed, $\operatorname{Var}(\hat\tau) \to \sigma^2\rho(1/J_1 + 1/J_0)$ .

Show solution

Step 1. $(1 - \rho)/m \to 0$ , so the cluster-mean variance tends to $\rho\sigma^2$ .

Step 2. Multiply by $1/J_1 + 1/J_0$ . $\blacksquare$

Even infinitely large clusters leave the between-cluster noise: only more clusters remove it.

Exercise 9 · Cluster means analysis

With equal cluster sizes, show that the individual-level difference in means equals the difference in the averages of cluster means. What changes with unequal sizes?

Show solution

Step 1. With $m$ pupils per cluster, the treated individual mean is $\frac{1}{mJ_1}\sum_g\sum_iY_{ig} = \frac{1}{J_1}\sum_g\bar Y_g$ , the average of cluster means. The same holds for controls.

Step 2. With unequal sizes, the individual mean weights clusters by size, while the average of cluster means weights them equally.

Answer: they then estimate different estimands: the effect for the average person versus the average cluster. Pre-specify which one you want.

Exercise 10 · Ignoring clustering

Show that an individual-level SE that ignores clustering is too small by the factor $\sqrt{\text{DEFF}}$ (equal cluster sizes, large samples).

Show solution

Step 1. The naive formula treats all $mJ$ observations as independent: $\sigma^2(1/(mJ_1) + 1/(mJ_0))$ .

Step 2. The true variance is DEFF times that (Exercise 7 and the theory block).

Step 3. Ratio of SEs: $\sqrt{\text{DEFF}}$ . $\blacksquare$

With DEFF = 3.9, naive SEs are about half the correct size and "significant" results can be spurious.

Part 3 · Applied

Exercise 11 · Planning a teacher-training trial

Teacher training is randomised across schools. Pupils' test scores are standardised, $m = 25$ pupils per school, ICC = 0.15. How many schools are needed for an MDE of 0.20 sd with 5% significance, 80% power and half the schools treated?

Show solution

Step 1. Variance factor: $\rho + (1 - \rho)/m = 0.15 + 0.85/25 = 0.184$ .

Step 2. With $P = 0.5$ , $P(1 - P)J = J/4$ , so $\text{MDE} = 2.80\sqrt{0.184 \times 4/J}$ .

Step 3. Set equal to 0.20: $(0.20/2.80)^2 = 0.736/J$ , i.e. $0.0051 = 0.736/J$ .

Step 4. $J = 0.736/0.0051 \approx 144.3$ ; round up to an even number.

Answer: about 146 schools (73 per arm), 3,650 pupils.

Exercise 12 · The value of a baseline test

In Exercise 11, school-average baseline scores explain 50% of the between-school variance. If you adjust for them, roughly how many schools are needed?

Show solution

Step 1. Between-school component falls from 0.15 to 0.075; the within-school part stays at 0.85/25 = 0.034 (ignoring individual-level adjustment).

Step 2. New factor: $0.075 + 0.034 = 0.109$ .

Step 3. $J = 0.109 \times 4/0.0051 \approx 85.5$ , so about 86 schools.

Answer: about 86 schools instead of 146: a cheap baseline test saves 60 schools.

Exercise 13 · Clinics of different sizes

A trial randomises 40 clinics, from 20 to 500 patients each, to a new hypertension protocol. The protocol may work better in large clinics. Which estimand does a patient-level comparison target, and which does a comparison of clinic means target?

Show solution

Step 1. Patient-level: each patient counts equally, so large clinics dominate: the effect for the average patient.

Step 2. Clinic-level: each clinic counts equally: the effect for the average clinic.

Step 3. If effects differ by size, these differ. A health ministry rolling out across clinics may care about patients; a manager supporting clinics may care about clinics.

Answer: state the estimand in the pre-analysis plan and weight accordingly.

Exercise 14 · Why not randomise individuals?

A village-level information campaign on safe water could instead be randomised by household. Give two reasons to randomise villages.

Show solution

Step 1. Spillovers: households talk to neighbours, so control households in treated villages would be partly treated, biasing the comparison toward zero (Day 32).

Step 2. Delivery: the campaign uses village meetings and posters, which cannot be withheld from some households.

Answer: cluster randomisation trades statistical power for a clean, policy-relevant comparison.

Exercise 15 · Few clusters

A district-level health reform is randomised across 8 districts (4 treated). What inference is appropriate?

Show solution

Step 1. Cluster-robust SEs need many clusters; with 8 they are unreliable (Day 29).

Step 2. Randomisation inference at the district level: $\binom84 = 70$ assignments, smallest two-sided p-value $2/70 \approx 0.029$ (Day 37).

Step 3. Alternatively, analyse district means with a t-test on 6 degrees of freedom, or the wild cluster bootstrap.

Answer: use district-level randomisation inference, and recognise that only large effects can be detected with 8 districts.

Verify in Python

import numpy as np
from scipy import stats

za, zb = stats.norm.ppf(0.975), stats.norm.ppf(0.80)
def mde(rho, m, J, P=0.5, sigma=1.0):
    return (za + zb) * np.sqrt(sigma**2 * (rho + (1 - rho) / m) / (P * (1 - P) * J))

print("Exercise 3 MDE:", round(mde(0.1, 30, 60), 3))
print("more pupils:", round(mde(0.1, 60, 60), 3), " more schools:", round(mde(0.1, 30, 120), 3))
J = 4 * 0.184 / (0.20 / (za + zb))**2
print("schools needed (Ex 11):", round(J, 1))

# Simulate a cluster RCT: naive versus correct SE
rng = np.random.default_rng(40)
J, m, rho = 60, 30, 0.1
est, naive_se = [], []
for _ in range(3000):
    u = rng.normal(0, np.sqrt(rho), J); T = np.repeat(rng.permutation(np.repeat([0, 1], J // 2)), m)
    y = np.repeat(u, m) + rng.normal(0, np.sqrt(1 - rho), J * m) + 0.0 * T
    est.append(y[T == 1].mean() - y[T == 0].mean())
    naive_se.append(np.sqrt(y[T == 1].var(ddof=1) / (J * m / 2) + y[T == 0].var(ddof=1) / (J * m / 2)))
print(f"true SD {np.std(est):.4f}, mean naive SE {np.mean(naive_se):.4f}, ratio {np.std(est) / np.mean(naive_se):.2f}"
      f" (sqrt DEFF = {np.sqrt(1 + 29 * 0.1):.2f})")

Key takeaways

Clustering inflates the variance by DEFF = 1 + (m − 1)ρ; ignoring it makes SEs too small by √DEFF.
Power comes mainly from the number of clusters; baseline covariates at the cluster level are very valuable.
With unequal cluster sizes, choose the estimand; with few clusters, use cluster-level randomisation inference.

Day 40. Cluster Randomisation & Design Effects

Theory

When to randomise clusters

The intra-cluster correlation

Variance and design effect

Power

Analysis

Part 1 · Warm-up

Exercise 1 · A design effect

Exercise 2 · Effective sample size

Exercise 3 · SE and MDE

Exercise 4 · More pupils or more schools?

Exercise 5 · Estimating the ICC

Part 2 · Proofs

Exercise 6 · Variance of a cluster mean

Exercise 7 · The design effect

Exercise 8 · The ceiling on cluster size

Exercise 9 · Cluster means analysis

Exercise 10 · Ignoring clustering

Part 3 · Applied

Exercise 11 · Planning a teacher-training trial

Exercise 12 · The value of a baseline test

Exercise 13 · Clinics of different sizes

Exercise 14 · Why not randomise individuals?

Exercise 15 · Few clusters

Verify in Python

Key takeaways

Simulation Lab: Cluster Designs & Power