Day 77: RD Validity: Density Tests & Covariate Balance

Theory

Why RD can work: imprecise control

Lee (2008): if units influence their running variable but cannot control it precisely (there is a random component with a continuous density), then the distribution of their characteristics, and hence of potential outcomes, is continuous at the cutoff. Units just above and just below are then as good as randomly assigned. If units can place themselves precisely on the preferred side (teachers rounding scores up to a pass mark, firms adjusting reported size to stay under a regulation threshold), those just above differ systematically from those just below.

Test 1: the density of the running variable

Under no manipulation, the density of $X$ is continuous at $c$ . McCrary (2008) and Cattaneo, Jansson and Ma estimate the density on each side by local polynomial methods and test

\theta = \ln f(c^+) - \ln f(c^-) = 0.

A spike just above $c$ and a dip just below indicate sorting. The test has no power against manipulation that moves units in both directions equally.

Test 2: covariate balance

Run the RD with pre-determined covariates (measured before treatment, or fixed: sex, prior scores, parental income) as outcomes. They should not jump. A jump means the units on either side differ in more than treatment.

Test 3: placebo cutoffs and outcomes

Estimate "effects" at fake cutoffs where nothing happens (for instance, the median of $X$ on each side), and effects on outcomes the treatment cannot affect. Both should be zero.

Robustness: donut RD

If manipulation or heaping (values piling up at round numbers, such as birth weights recorded at 1,500 g) is concentrated very close to $c$ , drop observations within a small "donut" around $c$ and re-estimate. Stability reassures; sensitivity reveals problems (at the cost of more extrapolation).

Other threats

Compound treatments (several policies switch at the same age), and sorting on the outcome side (selection into the sample that depends on crossing the cutoff).

Part 1 · Warm-up

Exercise 1 · A crude density check

Bins of equal width just below and just above a cutoff contain 480 and 620 observations. Compute the log density ratio and a rough z-statistic using $\operatorname{SE} \approx \sqrt{1/n_1 + 1/n_2}$ .

Show solution

Step 1. $\ln(620/480) = \ln 1.292 \approx 0.256$ .

Step 2. $\operatorname{SE} \approx \sqrt{1/480 + 1/620} = \sqrt{0.00208 + 0.00161} \approx 0.061$ .

Step 3. $z \approx 0.256/0.061 \approx 4.2$ .

Answer: strong evidence of excess mass just above the cutoff. (A proper test fits the density's slope on each side; this bin comparison ignores it.)

Exercise 2 · A covariate jumps

In an RD at a scholarship cutoff, the share of students with college-educated parents jumps by 6 points. What does this suggest?

Show solution

Step 1. Parental education is determined before the exam and cannot be caused by the scholarship.

Answer: students on either side differ; likely sorting (re-takes, appeals) by better-resourced families. The outcome jump may partly reflect background, not the scholarship.

Exercise 3 · Placebo cutoffs

At four placebo cutoffs, estimated jumps are 0.3, −0.5, 0.1 and 0.2 (each SE 0.4); at the true cutoff, 2.5 (SE 0.4). Interpret.

Show solution

Step 1. Placebo jumps are all within about 1.25 SE of zero.

Step 2. The real jump is more than 6 SE.

Answer: consistent with an effect specific to the cutoff, not a general wiggle in $E[Y \mid X]$ .

Exercise 4 · Why a donut?

Birth weights are often recorded at round numbers (1,500 g) and babies at those values have higher mortality for reasons unrelated to treatment. Why drop them?

Show solution

Step 1. Heaped observations are a different population (for example, poorer hospitals record rounded weights) concentrated exactly at the cutoff.

Step 2. They distort the estimates of both limits at $c$ .

Answer: excluding a small window around $c$ checks whether the estimate is driven by them.

Exercise 5 · Limits of the density test

Give an example where manipulation exists but the density test cannot detect it.

Show solution

Step 1. Some units push their score up across the cutoff while an equal number with different characteristics push theirs down (or are pushed down by an examiner).

Answer: the density stays smooth but the composition changes; covariate balance tests can still detect it.

Part 2 · Proofs

Exercise 6 · Imprecise control gives continuity

Let $X = X^* + \eta$ , where units choose $X^*$ (possibly based on their potential outcomes) and $\eta$ is independent noise with a continuous density $g$ . Show that the density of $X$ given any type of unit is continuous, so the share of each type is continuous at $c$ .

Show solution

Step 1. For a unit type $u$ with chosen $X^* = x^*_u$ , the density of $X$ is $g(x - x^*_u)$ , continuous in $x$ .

Step 2. Mixing over types: $f(x \mid u)$ continuous, and $P(u \mid X = x) = f(x \mid u)P(u)/f(x)$ is continuous wherever $f(x) > 0$ .

Step 3. So the composition of units, and $E[Y(d) \mid X = x]$ , is continuous at $c$ . $\blacksquare$

Exercise 7 · SE of a log count ratio

With independent Poisson counts $n_1, n_2$ , show $\operatorname{Var}(\ln n_2 - \ln n_1) \approx 1/n_1 + 1/n_2$ .

Show solution

Step 1. Delta method: $\operatorname{Var}(\ln n) \approx \operatorname{Var}(n)/E[n]^2 = E[n]/E[n]^2 = 1/E[n]$ for a Poisson count.

Step 2. Independence: add the two variances. $\blacksquare$

Exercise 8 · The bias from sorting

Suppose a share $s$ of units just above the cutoff are "movers" who would naturally have been just below, with mean untreated outcome $\mu_0 + \Delta$ instead of $\mu_0$ , and the effect is $\tau$ for everyone. Show the RD estimate converges to $\tau + s\Delta$ if movers come from a thin band just below (whose loss does not change the left limit).

Show solution

Step 1. Right limit: $(1 - s)(\mu_0 + \tau) + s(\mu_0 + \Delta + \tau) = \mu_0 + \tau + s\Delta$ .

Step 2. Left limit: $\mu_0$ (by assumption).

Step 3. Difference: $\tau + s\Delta$ . $\blacksquare$ Positive selection of movers ( $\Delta > 0$ ) overstates the effect.

Exercise 9 · Covariate balance is implied by continuity

Show that if the composition of units is continuous at $c$ (Exercise 6), then $E[W \mid X = x]$ is continuous at $c$ for every pre-determined $W$ .

Show solution

Step 1. $E[W \mid X = x] = \sum_u E[W \mid u]P(u \mid X = x)$ , with $W$ fixed for each unit type.

Step 2. Each $P(u \mid X = x)$ is continuous, so the sum is. $\blacksquare$ A jump in $E[W \mid X]$ therefore contradicts the continuity assumption that justifies the RD.

Exercise 10 · Two-sided manipulation with a smooth density

Construct an example in which the density of $X$ is continuous at $c$ but the composition jumps.

Show solution

Step 1. Start with a uniform $X$ for two equal-sized groups, A and B.

Step 2. A teacher moves A students from $[c - \epsilon, c)$ to $[c, c + \epsilon)$ , and an equal number of B students from $[c, c + \epsilon)$ to $[c - \epsilon, c)$ .

Step 3. Total counts on each side are unchanged, but just above $c$ there are only A students and just below only B students.

Answer: the density test passes while covariate balance fails. $\blacksquare$

Part 3 · Applied

Exercise 11 · Very low birth weight and heaping

A study finds that babies just below 1,500 g (who get extra care) have lower mortality than those just above. Re-analysis shows a spike of babies recorded at exactly 1,500 g, with high mortality, and that dropping them halves the estimated effect. What do you conclude?

Show solution

Step 1. Heaped babies at exactly 1,500 g fall on the "above" side and have high mortality for reasons unrelated to care (they come from hospitals with poorer measurement and care).

Step 2. They inflate the right limit, exaggerating the benefit of extra care.

Answer: the donut estimate is more credible; the original effect was partly an artefact of heaping.

Exercise 12 · Exam scores just at the pass mark

On a state exam with a pass mark of 65, 900 students score 63–64 and 2,100 score 65–66 (equal-width bins, otherwise smooth distribution). Interpret, and say what this means for an RD of passing on later outcomes.

Show solution

Step 1. $\ln(2{,}100/900) \approx 0.85$ , SE $\approx \sqrt{1/900 + 1/2{,}100} \approx 0.040$ : $z \approx 21$ .

Step 2. Graders bumped students just below 65 up to the pass mark.

Answer: the RD at 65 is invalid as is: students just above include bumped-up students who differ from others. Possible fixes: donut RD, or using an instrument based on the unmanipulated score.

Exercise 13 · Close elections

A study of close US House races finds that the party holding the seat wins close elections more often than chance, and that bare winners have more campaign money. What does this do to the incumbency RD?

Show solution

Step 1. Pre-determined covariates (prior incumbency, money) jump at the 0 margin: covariate balance fails.

Step 2. Some precise control over close outcomes (recounts, resources, local knowledge) breaks the as-good-as-random comparison.

Answer: the incumbency effect may be overstated; studies respond with covariate adjustment, other electoral contexts or narrower windows, and debate persists.

Exercise 14 · Sorting in a simulation

In the simulation below, 60% of units just below the cutoff with above-average untreated outcomes move just above it. Compare the RD estimate, the density test and a donut estimate with the true effect of 3.

Show solution

Step 1. The density test detects excess mass above the cutoff (log ratio about 0.64, $z > 10$ ).

Step 2. The RD estimate is biased upwards (about 3.65) because movers have higher untreated outcomes (Exercise 8).

Step 3. Dropping a 0.05-wide donut around the cutoff brings the estimate back near 3 (about 2.8), with more noise.

Exercise 15 · A validity checklist

Write a checklist of validity checks for an RD estimating the effect of a means-tested child benefit (income cutoff) on child health.

Show solution

Step 1. Institutional knowledge: can families adjust reported income to qualify? Does anything else switch at the cutoff?

Step 2. Density test of reported income at the cutoff.

Step 3. Covariate balance: parental education, household size, child birth weight.

Step 4. Placebo cutoffs and outcomes measured before the benefit (baseline health).

Step 5. Donut RD and bandwidth sensitivity; RBC intervals.

Verify in Python

import numpy as np

z = lambda n1, n2: np.log(n2 / n1) / np.sqrt(1 / n1 + 1 / n2)
print(f"Ex 1: log ratio {np.log(620 / 480):.3f}, z {z(480, 620):.1f} | Ex 12: log ratio {np.log(2100 / 900):.2f}, z {z(900, 2100):.1f}")

rng = np.random.default_rng(77); n = 40_000
x = rng.uniform(-1, 1, n); y0 = 10 + 4 * x + rng.normal(0, 2, n)
mover = (x < 0) & (x > -.05) & (y0 - 10 - 4 * x > 0) & (rng.random(n) < .6)
xm = x.copy(); xm[mover] = rng.uniform(0, .05, mover.sum())
y = y0 + 3 * (xm >= 0)
def rd(x, y, h=.3):
    def side(xx, yy):
        w = np.clip(1 - np.abs(xx) / h, 0, None); m = w > 0
        X = np.column_stack([np.ones(m.sum()), xx[m]]); W = w[m]
        return np.linalg.solve(X.T @ (W[:, None] * X), X.T @ (W * yy[m]))[0]
    r = x >= 0
    return side(x[r], y[r]) - side(-x[~r], y[~r])
nl, nr = ((xm < 0) & (xm >= -.05)).sum(), ((xm >= 0) & (xm < .05)).sum()
print(f"movers {mover.sum()}; density bins {nl} vs {nr}: log ratio {np.log(nr / nl):.2f}, z {z(nl, nr):.1f}")
keep = np.abs(xm) > .05
print(f"RD with sorting {rd(xm, y):.2f}  donut {rd(xm[keep], y[keep]):.2f}  no sorting {rd(x, y0 + 3 * (x >= 0)):.2f}  (truth 3)")

Key takeaways

RD is credible when units cannot precisely control which side of the cutoff they land on.
Check the density of the running variable, balance of pre-determined covariates, placebo cutoffs and outcomes, and robustness to a donut.
Sorting by units with different potential outcomes biases the jump; two-sided manipulation can hide from density tests but not from covariate checks.

Day 77. RD Validity: Density Tests & Covariate Balance

Theory

Why RD can work: imprecise control

Test 1: the density of the running variable

Test 2: covariate balance

Test 3: placebo cutoffs and outcomes

Robustness: donut RD

Other threats

Part 1 · Warm-up

Exercise 1 · A crude density check

Exercise 2 · A covariate jumps

Exercise 3 · Placebo cutoffs

Exercise 4 · Why a donut?

Exercise 5 · Limits of the density test

Part 2 · Proofs

Exercise 6 · Imprecise control gives continuity

Exercise 7 · SE of a log count ratio

Exercise 8 · The bias from sorting

Exercise 9 · Covariate balance is implied by continuity

Exercise 10 · Two-sided manipulation with a smooth density

Part 3 · Applied

Exercise 11 · Very low birth weight and heaping

Exercise 12 · Exam scores just at the pass mark

Exercise 13 · Close elections

Exercise 14 · Sorting in a simulation

Exercise 15 · A validity checklist

Verify in Python

Key takeaways

Simulation Lab: Manipulation & RD Validity