Day 12: The Delta Method · Math for Causal Inference

Theory

The idea

You have an estimate $\hat\theta$ with $\sqrt n(\hat\theta - \theta) \xrightarrow{d} N(0, \sigma^2)$ , but you care about $g(\theta)$ : a percentage effect, an elasticity, a ratio. A first-order Taylor expansion around $\theta$ gives

g(\hat\theta) \approx g(\theta) + g'(\theta)(\hat\theta - \theta),

so $g(\hat\theta)$ is approximately a linear function of an approximately normal variable:

\sqrt n\big(g(\hat\theta) - g(\theta)\big) \xrightarrow{d} N\big(0,\; g'(\theta)^2\sigma^2\big), \qquad \operatorname{SE}\big(g(\hat\theta)\big) \approx |g'(\hat\theta)|\,\operatorname{SE}(\hat\theta).

Several estimates at once

For a vector $\hat\theta$ with covariance matrix $\Sigma$ and a scalar function $g$ with gradient $\nabla g$ :

\operatorname{Var}\big(g(\hat\theta)\big) \approx \nabla g(\theta)^{\top}\,\Sigma\,\nabla g(\theta).

For a ratio $R = A/B$ :

\operatorname{Var}(\hat R) \approx \frac{\operatorname{Var}(\hat A)}{B^2} + \frac{A^2\operatorname{Var}(\hat B)}{B^4} - \frac{2A\operatorname{Cov}(\hat A, \hat B)}{B^3}.

Limits

It is a local linear approximation: poor when $g$ is very curved relative to the SE, or when $\hat\theta$ is near a point where $g$ blows up (a ratio with denominator near zero, e.g. a weak first stage).
If $g'(\theta) = 0$ the first-order term vanishes and a second-order expansion is needed.
The bootstrap (Day 9) is the usual cross-check.

Part 1 · Warm-up

Exercise 1 · Derivative of the log

What is $g'(x)$ for $g(x) = \ln x$ , and what does it imply for $\operatorname{SE}(\ln\hat\theta)$ ?

Show solution

$g'(x) = 1/x$ , so $\operatorname{SE}(\ln\hat\theta) \approx \operatorname{SE}(\hat\theta)/\hat\theta$ : the coefficient of variation.

Exercise 2 · Log of a mean

$\bar X = 50$ with SE 2. Find the SE of $\ln\bar X$ .

Show solution

$2/50 = 0.04$ .

Exercise 3 · Square root

$\hat\theta = 16$ with SE 0.8. Find the SE of $\sqrt{\hat\theta}$ .

Show solution

$g'(x) = 1/(2\sqrt x) = 1/8$ , so SE $\approx 0.8/8 = 0.1$ .

Exercise 4 · Inverse

$\hat\theta = 4$ with SE 0.2. Find the SE of $1/\hat\theta$ .

Show solution

$|g'(x)| = 1/x^2 = 1/16$ , so SE $\approx 0.2/16 = 0.0125$ .

Exercise 5 · Odds

$\hat p = 0.6$ with SE 0.0894 (Day 11). Find the SE of the odds $\hat p/(1 - \hat p)$ .

Show solution

$g'(p) = 1/(1 - p)^2 = 1/0.16 = 6.25$ , so SE $\approx 6.25 \times 0.0894 \approx 0.559$ around the odds estimate 1.5.

Part 2 · Proofs

Exercise 6 · Deriving the delta method

Using a Taylor expansion with remainder and the CLT, derive the univariate delta method.

Show solution

Step 1 (mean value theorem). For some $\tilde\theta$ between $\hat\theta$ and $\theta$ :

g(\hat\theta) = g(\theta) + g'(\tilde\theta)(\hat\theta - \theta).

Step 2. Rearrange and multiply by $\sqrt n$ :

\sqrt n\big(g(\hat\theta) - g(\theta)\big) = g'(\tilde\theta)\cdot\sqrt n(\hat\theta - \theta).

Step 3. $\hat\theta \xrightarrow{p} \theta$ and $\tilde\theta$ is squeezed between them, so $\tilde\theta \xrightarrow{p} \theta$ . With $g'$ continuous, $g'(\tilde\theta) \xrightarrow{p} g'(\theta)$ .

Step 4. By assumption (the CLT), $\sqrt n(\hat\theta - \theta) \xrightarrow{d} N(0, \sigma^2)$ .

Step 5. Slutsky's theorem (a term converging to a constant times a term converging in distribution):

\sqrt n\big(g(\hat\theta) - g(\theta)\big) \xrightarrow{d} g'(\theta)\,N(0, \sigma^2) = N\big(0,\; g'(\theta)^2\sigma^2\big). \qquad\blacksquare

Exercise 7 · The ratio formula

Derive the variance of $\hat A/\hat B$ from the multivariate delta method.

Show solution

$g(A, B) = A/B$ has gradient $\nabla g = (1/B,\; -A/B^2)$ . Then

\nabla g^{\top}\Sigma\nabla g = \frac{\operatorname{Var}(\hat A)}{B^2} + \frac{A^2}{B^4}\operatorname{Var}(\hat B) - \frac{2A}{B^3}\operatorname{Cov}(\hat A, \hat B).

Exercise 8 · Second-order bias

Using a second-order expansion, show $E[g(\bar X)] \approx g(\mu) + \tfrac12 g''(\mu)\sigma^2/n$ , and apply it to $\ln\bar X$ .

Show solution

Step 1. $g(\bar X) \approx g(\mu) + g'(\mu)(\bar X - \mu) + \tfrac12 g''(\mu)(\bar X - \mu)^2$ .

Step 2. Take expectations: the linear term has mean zero and $E[(\bar X - \mu)^2] = \sigma^2/n$ .

Step 3. For $\ln$ : $g'' = -1/\mu^2$ , so $E[\ln\bar X] \approx \ln\mu - \sigma^2/(2n\mu^2)$ .

Step 4. The log of a mean is biased downward, as Jensen's inequality predicts, but the bias vanishes at rate $1/n$ .

Exercise 9 · When the first derivative is zero

Let $\bar X$ be a mean with $\mu = 0$ and variance $\sigma^2/n$ , and $g(x) = x^2$ . Why does the delta method fail, and what is the limit of $n\bar X^2/\sigma^2$ ?

Show solution

Step 1. $g'(0) = 0$ , so the first-order approximation predicts zero variance, which is wrong.

Step 2. Instead, $\sqrt n\bar X/\sigma \xrightarrow{d} N(0, 1)$ , so $n\bar X^2/\sigma^2 \xrightarrow{d} \chi^2_1$ .

Step 3. The estimator converges at rate $n$ , not $\sqrt n$ , and its distribution is not normal.

Exercise 10 · Log-odds

Derive the SE of the log-odds $\ln(\hat p/(1 - \hat p))$ and evaluate it at $\hat p = 0.6$ , $n = 30$ .

Show solution

$g'(p) = 1/p + 1/(1 - p) = 1/(p(1 - p))$ . With $\operatorname{SE}(\hat p) = \sqrt{p(1 - p)/n}$ :

\operatorname{SE} = \frac{1}{\sqrt{n\,p(1 - p)}} = \frac{1}{\sqrt{30 \times 0.24}} \approx 0.373.

Part 3 · Applied

Exercise 11 · From log points to percent

In a log-wage regression, a training dummy has $\hat\beta = 0.08$ (SE 0.02). Report the exact percentage effect $100(e^{\beta} - 1)$ with its SE.

Show solution

Step 1. Exact percentage effect: $g(\beta) = 100(e^{\beta} - 1) = 100(e^{0.08} - 1) \approx 8.33\%$ .

Step 2. Derivative: $g'(\beta) = 100e^{\beta} = 108.33$ .

Step 3. Delta method: $\operatorname{SE} \approx 108.33 \times 0.02 \approx 2.17$ percentage points.

Answer: training raises wages by 8.3% (SE 2.2 points).

Exercise 12 · Cost-effectiveness ratio

A programme costs an extra $\Delta C = 1{,}200$ (SE 300) and adds $\Delta E = 0.4$ QALYs (SE 0.1), estimated independently. Find the ICER and its SE.

Show solution

Step 1. Point estimate: $\text{ICER} = 1200/0.4 = 3{,}000$ per QALY.

Step 2. Ratio formula with zero covariance, $A = 1200$ , $B = 0.4$ :

\operatorname{Var} \approx \frac{\operatorname{Var}(\hat A)}{B^2} + \frac{A^2\operatorname{Var}(\hat B)}{B^4}.

Step 3. First term: $300^2/0.4^2 = 90{,}000/0.16 = 562{,}500$ .

Step 4. Second term: $1200^2 \times 0.1^2/0.4^4 = 1{,}440{,}000 \times 0.01/0.0256 = 562{,}500$ .

Step 5. Total $1{,}125{,}000$ , so $\operatorname{SE} \approx 1{,}061$ .

Answer: $3,000 per QALY (SE ≈ $1,061). The imprecise denominator contributes half the variance; a bootstrap interval would be visibly skewed.

Exercise 13 · A relative treatment effect

The effect is $\hat\tau = 5$ (SE 1) and the control mean is $\hat\mu_0 = 50$ (SE 2), independent. Find the relative effect $\hat\tau/\hat\mu_0$ and its SE.

Show solution

Step 1. Point estimate: $R = 5/50 = 0.10$ .

Step 2. Ratio formula with $A = 5$ , $B = 50$ , independent estimates:

\operatorname{Var} \approx \frac{1^2}{50^2} + \frac{5^2 \times 2^2}{50^4} = 0.0004 + 0.000016 = 0.000416.

Step 3. $\operatorname{SE} = \sqrt{0.000416} \approx 0.0204$ .

Answer: a 10% relative effect, ± about 2 points. Almost all the uncertainty comes from $\hat\tau$ .

Exercise 14 · The Wald (IV) estimator

An encouragement design has reduced form 0.6 (SE 0.2) and first stage 0.3 (SE 0.05), independent. Find the IV estimate and its SE.

Show solution

Step 1. Point estimate: $\hat\beta = 0.6/0.3 = 2$ .

Step 2. Ratio formula with $A = 0.6$ (reduced form), $B = 0.3$ (first stage):

\frac{0.2^2}{0.3^2} = \frac{0.04}{0.09} \approx 0.444, \qquad \frac{0.6^2 \times 0.05^2}{0.3^4} = \frac{0.36 \times 0.0025}{0.0081} \approx 0.111.

Step 3. $\operatorname{Var} \approx 0.556$ , $\operatorname{SE} \approx 0.745$ .

Answer: IV estimate 2 (SE ≈ 0.75). With a weak first stage (denominator near zero) this approximation breaks down: the weak-instruments problem of Phase 5.

Exercise 15 · Log of mean income: delta method versus simulation

A survey estimates mean household income (in $10,000) as $\bar X$ with SE 2. Analysts report $\ln\bar X$ . Compare the delta-method SE of $\ln\bar X$ with a simulation when the true mean is 50 (a rich region) and when it is 5 (a poor region). When does the delta method fail?

Show solution

Step 1. Delta method: $\operatorname{SE}(\ln\bar X) \approx \operatorname{SE}/\mu$ .

Step 2. Rich region: $2/50 = 0.04$ . Poor region: $2/5 = 0.40$ .

Step 3. Simulation (code below): about 0.040 in the rich region, but about 0.52 in the poor region.

Step 4. Reason: when $\operatorname{SE}/\mu$ is large, $\bar X$ ranges over values where $\ln$ is strongly curved, so the linear approximation is poor and $\ln\bar X$ is skewed.

Answer: the delta method is excellent when the coefficient of variation is small (0.04) and misleading when it is large (0.4); check with the bootstrap or simulation.

Verify in Python

import numpy as np

rng = np.random.default_rng(12)
for mu, se in ((50, 2), (5, 2)):
    xbar = rng.normal(mu, se, 1_000_000)
    xbar = xbar[xbar > 0]
    print(f"mu={mu}, SE={se}: delta SE of ln = {se/mu:.4f}, simulated = {np.log(xbar).std():.4f}")

# ICER (Exercise 12): delta method vs simulation
dC = rng.normal(1200, 300, 1_000_000); dE = rng.normal(0.4, 0.1, 1_000_000)
icer = dC / dE
q25, q50, q75 = np.percentile(icer, [25, 50, 75])
print("ICER delta SE ~ 1061; simulated robust SD (IQR/1.349):", round((q75 - q25) / 1.349),
      "; simulated 2.5/50/97.5 percentiles:", np.percentile(icer, [2.5, 50, 97.5]).round())
# (the plain SD is unstable: a few draws of dE near zero make the ratio explode)

print("percent effect:", round(100 * (np.exp(0.08) - 1), 2), "SE", round(100 * np.exp(0.08) * 0.02, 2))

Key takeaways

$\operatorname{SE}(g(\hat\theta)) \approx |g'(\hat\theta)|\operatorname{SE}(\hat\theta)$ ; for several estimates use the gradient and the covariance matrix.
Ratios with noisy denominators (ICERs, IV with a weak first stage) are where the approximation is weakest.
Check the delta method against the bootstrap when the transformation is strongly non-linear.

Day 12. The Delta Method

Theory

The idea

Several estimates at once

Limits

Part 1 · Warm-up

Exercise 1 · Derivative of the log

Exercise 2 · Log of a mean

Exercise 3 · Square root

Exercise 4 · Inverse

Exercise 5 · Odds

Part 2 · Proofs

Exercise 6 · Deriving the delta method

Exercise 7 · The ratio formula

Exercise 8 · Second-order bias

Exercise 9 · When the first derivative is zero

Exercise 10 · Log-odds

Part 3 · Applied

Exercise 11 · From log points to percent

Exercise 12 · Cost-effectiveness ratio

Exercise 13 · A relative treatment effect

Exercise 14 · The Wald (IV) estimator

Exercise 15 · Log of mean income: delta method versus simulation

Verify in Python

Key takeaways

Simulation Lab: Delta Method vs Simulation