Day 6: Conditional Expectation, the LIE & Total Variance

Theory

The conditional expectation function

$E[Y \mid X = x]$ is the mean of $Y$ among units with $X = x$ . As a function of $x$ it is the conditional expectation function (CEF), $m(x)$ . Plugging in the random $X$ gives $E[Y \mid X] = m(X)$ , itself a random variable.

Three workhorse results

1. Law of iterated expectations (LIE).

E[Y] = E\big[E[Y \mid X]\big] = \sum_x E[Y \mid X = x]\,P(X = x).

The overall mean is the average of the group means, weighted by group size.

2. Taking out what is known. For any function $g$ , $E[g(X)\,Y \mid X] = g(X)\,E[Y \mid X]$ .

3. CEF decomposition. Write $Y = m(X) + e$ with $e = Y - m(X)$ . Then $E[e \mid X] = 0$ , so $e$ is uncorrelated with every function of $X$ . Moreover, $m(X)$ is the best predictor of $Y$ : it minimises $E[(Y - g(X))^2]$ over all functions $g$ . If the CEF happens to be linear, $m(x) = a + bx$ , then $b = \operatorname{Cov}(X, Y)/\operatorname{Var}(X)$ , the regression slope.

Law of total variance

\operatorname{Var}(Y) = \underbrace{E\big[\operatorname{Var}(Y \mid X)\big]}_{\text{within groups}} + \underbrace{\operatorname{Var}\big(E[Y \mid X]\big)}_{\text{between groups}}.

The share of variance "explained" by $X$ is the between part divided by the total: the population $R^2$ of the CEF.

The adjustment formula

Suppose treatment is as good as random within levels of $X$ : $\big(Y(1), Y(0)\big) \perp T \mid X$ ("unconfoundedness"). Then the LIE gives

E[Y(1)] = \sum_x E[Y \mid T = 1, X = x]\,P(X = x),

and similarly for $Y(0)$ . Weighting by $P(X = x \mid T = 1)$ instead gives the effect on the treated (ATT). This adjustment formula (also called standardisation or the g-formula) is the backbone of matching, regression adjustment and inverse probability weighting.

Part 1 · Warm-up

Exercise 1 · Average wage by education

60% of workers have high school (mean wage $40k) and 40% college (mean wage $70k). Find the average wage.

Show solution

By the LIE, $E[W] = 0.6(40) + 0.4(70) = 24 + 28 = 52$ thousand.

Exercise 2 · Checking the LIE

Using Day 5's table, $E[Y \mid X = 1] = 0.75$ , $E[Y \mid X = 0] = 0.50$ and $P(X = 1) = 0.4$ . Verify that the LIE gives $E[Y] = 0.6$ .

Show solution

$E[Y] = 0.4(0.75) + 0.6(0.50) = 0.30 + 0.30 = 0.60$ . ✓

Exercise 3 · Conditional variances

In the same table, compute $\operatorname{Var}(Y \mid X = 1)$ and $\operatorname{Var}(Y \mid X = 0)$ .

Show solution

$Y \mid X$ is Bernoulli: $\operatorname{Var}(Y \mid X = 1) = 0.75 \times 0.25 = 0.1875$ and $\operatorname{Var}(Y \mid X = 0) = 0.5 \times 0.5 = 0.25$ .

Exercise 4 · Total variance

High-school wages have mean 40 and variance 100; college wages have mean 70 and variance 400; the shares are 0.6 and 0.4. Find $\operatorname{Var}(W)$ via the law of total variance.

Show solution

Step 1. Within-group part, the share-weighted average of group variances:

E[\operatorname{Var}(W \mid X)] = 0.6(100) + 0.4(400) = 60 + 160 = 220.

Step 2. Between-group part, the variance of the group means. $E[W] = 52$ (Exercise 1), so

\operatorname{Var}(E[W \mid X]) = 0.6(40 - 52)^2 + 0.4(70 - 52)^2 = 0.6(144) + 0.4(324) = 86.4 + 129.6 = 216.

(Shortcut for two groups: $0.6 \times 0.4 \times 30^2 = 216$ .)

Step 3. Total: $\operatorname{Var}(W) = 220 + 216 = 436$ .

Answer: 436 (sd ≈ $20.9k).

Exercise 5 · Independence

Show that if $X$ and $Y$ are independent, $E[Y \mid X] = E[Y]$ .

Show solution

Independence means $p(y \mid x) = p_Y(y)$ , so $E[Y \mid X = x] = \sum_y y\,p_Y(y) = E[Y]$ for every $x$ . The CEF is flat.

Part 2 · Proofs

Exercise 6 · Proving the LIE

Prove $E[E[Y \mid X]] = E[Y]$ for discrete variables.

Show solution

E\big[E[Y \mid X]\big] = \sum_x\Big(\sum_y y\,\frac{p(x, y)}{p_X(x)}\Big)p_X(x) = \sum_x\sum_y y\,p(x, y) = \sum_y y\,p_Y(y) = E[Y].

Exercise 7 · The CEF error is orthogonal to functions of X

With $e = Y - E[Y \mid X]$ , show that $E[e \mid X] = 0$ and $E[g(X)\,e] = 0$ for any $g$ .

Show solution

$E[e \mid X] = E[Y \mid X] - E[Y \mid X] = 0$ . Then, by the LIE and "taking out what is known",

E[g(X)e] = E\big[E[g(X)e \mid X]\big] = E\big[g(X)\,E[e \mid X]\big] = 0.

Exercise 8 · The CEF is the best predictor

Prove that $E[(Y - g(X))^2] \ge E[(Y - m(X))^2]$ for any function $g$ .

Show solution

Step 1. Write $Y - g(X) = e + \big(m(X) - g(X)\big)$ .

Step 2. Expanding,

E[(Y - g(X))^2] = E[e^2] + 2E\big[e\,(m(X) - g(X))\big] + E\big[(m(X) - g(X))^2\big].

Step 3. The middle term is zero by Exercise 7 (it is $e$ times a function of $X$ ), and the last is non-negative.

Step 4. So the error is smallest when $g = m$ .

Exercise 9 · Proving total variance

Prove $\operatorname{Var}(Y) = E[\operatorname{Var}(Y \mid X)] + \operatorname{Var}(E[Y \mid X])$ .

Show solution

Write $m(X) = E[Y \mid X]$ .

Step 1. Conditional variance shortcut: $\operatorname{Var}(Y \mid X) = E[Y^2 \mid X] - m(X)^2$ .

Step 2. Take expectations and use the LIE on the first term:

E[\operatorname{Var}(Y \mid X)] = E[Y^2] - E[m(X)^2].

Step 3. Variance of the CEF, using $E[m(X)] = E[Y]$ (LIE):

\operatorname{Var}(m(X)) = E[m(X)^2] - (E[Y])^2.

Step 4. Add steps 2 and 3; the $E[m(X)^2]$ terms cancel:

E[\operatorname{Var}(Y \mid X)] + \operatorname{Var}(m(X)) = E[Y^2] - (E[Y])^2 = \operatorname{Var}(Y). \qquad\blacksquare

Exercise 10 · A linear CEF is the regression line

Suppose $E[Y \mid X] = a + bX$ . Prove $b = \operatorname{Cov}(X, Y)/\operatorname{Var}(X)$ .

Show solution

By the LIE, $E[XY] = E[X\,E[Y \mid X]] = E[X(a + bX)] = aE[X] + bE[X^2]$ and $E[Y] = a + bE[X]$ . Then

\operatorname{Cov}(X, Y) = aE[X] + bE[X^2] - E[X](a + bE[X]) = b\operatorname{Var}(X).

Part 3 · Applied

A hospital's records give mean recovery rates $E[Y \mid T, S]$ by treatment $T$ and severity $S$ . In the population, 70% of cases are mild.

	Mild	Severe
Treated ( $T = 1$ )	0.90	0.60
Untreated ( $T = 0$ )	0.80	0.40

Assume treatment is as good as random within severity level.

Exercise 11 · The adjustment formula

Compute $E[Y(1)]$ , $E[Y(0)]$ and the ATE.

Show solution

Step 1. Unconfoundedness lets us use the treated patients' recovery rate in each severity group as $E[Y(1) \mid S]$ . Average over the population case mix (70% mild):

E[Y(1)] = 0.7(0.90) + 0.3(0.60) = 0.63 + 0.18 = 0.81.

Step 2. Same for the untreated:

E[Y(0)] = 0.7(0.80) + 0.3(0.40) = 0.56 + 0.12 = 0.68.

Step 3. ATE $= 0.81 - 0.68 = 0.13$ .

Answer: treatment raises the recovery rate by 13 percentage points on average.

Exercise 12 · What the naive comparison says

Doctors treat the sickest: 60% of treated patients are severe, but only 10% of untreated ones. Compute the naive difference $E[Y \mid T = 1] - E[Y \mid T = 0]$ .

Show solution

Step 1. Treated patients' case mix: 40% mild, 60% severe. Their average recovery:

E[Y \mid T = 1] = 0.4(0.90) + 0.6(0.60) = 0.36 + 0.36 = 0.72.

Step 2. Untreated case mix: 90% mild, 10% severe:

E[Y \mid T = 0] = 0.9(0.80) + 0.1(0.40) = 0.72 + 0.04 = 0.76.

Step 3. Naive difference: $0.72 - 0.76 = -0.04$ .

Answer: the raw comparison says treatment lowers recovery by 4 points, although it raises it by 13. Each group is averaged over its own case mix, and the treated are much sicker.

Exercise 13 · ATT versus ATE

Compute the effect on the treated, $E[Y(1) - Y(0) \mid T = 1]$ . Why does it differ from the ATE?

Show solution

Step 1. Effect within each stratum: mild $0.90 - 0.80 = 0.10$ ; severe $0.60 - 0.40 = 0.20$ .

Step 2. ATT averages these over the treated case mix (40% mild, 60% severe):

\text{ATT} = 0.4(0.10) + 0.6(0.20) = 0.04 + 0.12 = 0.16.

Step 3. ATE uses the population mix (70% mild): $0.7(0.10) + 0.3(0.20) = 0.13$ .

Answer: ATT = 0.16 > ATE = 0.13, because the effect is larger for severe cases and the treated are disproportionately severe.

Exercise 14 · Why a voucher lottery works

A city allocates private-school vouchers by lottery: $T = 1$ for winners. Let $Y(1)$ be a child's test score with a voucher. Show that if winning is independent of $Y(1)$ , then the winners' average score identifies $E[Y(1)]$ , i.e. $E[Y \mid T = 1] = E[Y(1)]$ .

Show solution

Step 1. Among winners we observe the voucher outcome: $Y = Y(1)$ when $T = 1$ , so $E[Y \mid T = 1] = E[Y(1) \mid T = 1]$ .

Step 2. The lottery makes $T$ independent of $Y(1)$ , so conditioning on $T$ does not change its mean (Exercise 5): $E[Y(1) \mid T = 1] = E[Y(1)]$ .

Step 3. The same argument for losers gives $E[Y \mid T = 0] = E[Y(0)]$ .

Answer: the winners' mean minus the losers' mean equals $E[Y(1)] - E[Y(0)]$ , the ATE of a voucher. That is why lotteries are prized in education research.

Exercise 15 · How much does education explain?

Using Exercise 4, what share of wage variance is explained by education group?

Show solution

Step 1. From Exercise 4: between-group variance 216, total 436.

Step 2. Share explained $= 216/436 \approx 0.495$ .

Answer: education group explains about half the variance in wages (the population $R^2$ of the CEF); the other half is variation among workers with the same education.

Verify in Python

import numpy as np

mild = 0.7
cef = {(1, "mild"): 0.90, (1, "severe"): 0.60, (0, "mild"): 0.80, (0, "severe"): 0.40}
w_pop = {"mild": mild, "severe": 1 - mild}
w_treated = {"mild": 0.4, "severe": 0.6}
w_untreated = {"mild": 0.9, "severe": 0.1}

def mean(t, w):
    return sum(cef[(t, s)] * w[s] for s in w)

print("ATE  :", round(mean(1, w_pop) - mean(0, w_pop), 3))
print("ATT  :", round(mean(1, w_treated) - mean(0, w_treated), 3))
print("naive:", round(mean(1, w_treated) - mean(0, w_untreated), 3))

# Law of total variance by simulation (Exercise 4)
rng = np.random.default_rng(3)
n = 1_000_000
college = rng.random(n) < 0.4
W = np.where(college, rng.normal(70, 20, n), rng.normal(40, 10, n))
within = 0.6 * W[~college].var() + 0.4 * W[college].var()
between = 0.6 * 0.4 * (W[college].mean() - W[~college].mean())**2
print(f"Var(W) = {W.var():.1f}; within + between = {within + between:.1f}")

Key takeaways

The overall mean is a weighted average of conditional means (LIE).
The CEF is the best possible predictor; its error is uncorrelated with every function of $X$ .
With unconfoundedness, the adjustment formula turns subgroup comparisons into ATE and ATT.

Day 6. Conditional Expectation, the LIE & Total Variance

Theory

The conditional expectation function

Three workhorse results

Law of total variance

The adjustment formula

Part 1 · Warm-up

Exercise 1 · Average wage by education

Exercise 2 · Checking the LIE

Exercise 3 · Conditional variances

Exercise 4 · Total variance

Exercise 5 · Independence

Part 2 · Proofs

Exercise 6 · Proving the LIE

Exercise 7 · The CEF error is orthogonal to functions of X

Exercise 8 · The CEF is the best predictor

Exercise 9 · Proving total variance

Exercise 10 · A linear CEF is the regression line

Part 3 · Applied

Exercise 11 · The adjustment formula

Exercise 12 · What the naive comparison says

Exercise 13 · ATT versus ATE

Exercise 14 · Why a voucher lottery works

Exercise 15 · How much does education explain?

Verify in Python

Key takeaways

Simulation Lab: Conditional Expectation & Total Variance