Day 5: Joint Distributions, Covariance & Correlation

Theory

Joint, marginal and conditional

For two discrete variables, the joint pmf is $p(x, y) = P(X = x, Y = y)$ . Summing over one variable gives the marginal, $p_X(x) = \sum_y p(x, y)$ , and dividing gives the conditional, $p(y \mid x) = p(x, y)/p_X(x)$ . $X$ and $Y$ are independent when $p(x, y) = p_X(x)p_Y(y)$ for all $x, y$ . Continuous variables work the same way with densities and integrals.

Covariance and correlation

\operatorname{Cov}(X, Y) = E\big[(X - \mu_X)(Y - \mu_Y)\big] = E[XY] - E[X]E[Y], \qquad \rho_{XY} = \frac{\operatorname{Cov}(X, Y)}{\sigma_X \sigma_Y} \in [-1, 1].

Covariance has units (dollars × years); correlation is unit-free. Useful rules:

$\operatorname{Cov}(X, X) = \operatorname{Var}(X)$
$\operatorname{Cov}(aX + b, cY + d) = ac\,\operatorname{Cov}(X, Y)$
$\operatorname{Var}(X \pm Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) \pm 2\operatorname{Cov}(X, Y)$
Independence ⇒ zero covariance, but not the other way round.

Covariance is the regression slope's numerator

The best linear predictor of $Y$ given $X$ has slope

\beta = \frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}.

If $Y = \beta X + \gamma U + \varepsilon$ and you leave $U$ out, the slope you estimate is $\beta + \gamma\,\operatorname{Cov}(X, U)/\operatorname{Var}(X)$ : the famous omitted-variable bias.

The bivariate normal

If $(X, Y)$ is bivariate normal with correlation $\rho$ , the conditional mean is linear and the conditional variance is constant:

E[Y \mid X = x] = \mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(x - \mu_X), \qquad \operatorname{Var}(Y \mid X) = \sigma_Y^2(1 - \rho^2).

Explore it in 3D in the Simulation Lab.

Part 1 · Warm-up

The joint distribution of job training $X$ and employment $Y$ is:

	$Y = 0$	$Y = 1$
$X = 0$	0.30	0.30
$X = 1$	0.10	0.30

Exercise 1 · Marginals

Find $P(X = 1)$ and $P(Y = 1)$ .

Show solution

$P(X = 1) = 0.10 + 0.30 = 0.40$ and $P(Y = 1) = 0.30 + 0.30 = 0.60$ .

Exercise 2 · The product moment

Compute $E[XY]$ .

Show solution

$XY = 1$ only when both are 1, so $E[XY] = P(X = 1, Y = 1) = 0.30$ .

Exercise 3 · Covariance

Compute $\operatorname{Cov}(X, Y)$ .

Show solution

$\operatorname{Cov}(X, Y) = 0.30 - 0.40 \times 0.60 = 0.06$ .

Exercise 4 · Correlation

Compute $\rho_{XY}$ .

Show solution

$\operatorname{Var}(X) = 0.4 \times 0.6 = 0.24$ and $\operatorname{Var}(Y) = 0.6 \times 0.4 = 0.24$ , so $\rho = 0.06/\sqrt{0.24 \times 0.24} = 0.25$ .

Exercise 5 · Conditionals and independence

Find $P(Y = 1 \mid X = 1)$ and $P(Y = 1 \mid X = 0)$ . Are $X$ and $Y$ independent?

Show solution

$P(Y = 1 \mid X = 1) = 0.30/0.40 = 0.75$ and $P(Y = 1 \mid X = 0) = 0.30/0.60 = 0.50$ . They differ, so $X$ and $Y$ are not independent (also, $0.30 \ne 0.40 \times 0.60$ ).

Part 2 · Proofs

Exercise 6 · Covariance shortcut

Prove $\operatorname{Cov}(X, Y) = E[XY] - E[X]E[Y]$ .

Show solution

E[(X - \mu_X)(Y - \mu_Y)] = E[XY] - \mu_Y E[X] - \mu_X E[Y] + \mu_X\mu_Y = E[XY] - \mu_X\mu_Y.

Exercise 7 · Independence implies zero covariance

Prove that if $X$ and $Y$ are independent (discrete), $\operatorname{Cov}(X, Y) = 0$ .

Show solution

E[XY] = \sum_x\sum_y xy\,p_X(x)p_Y(y) = \Big(\sum_x x\,p_X(x)\Big)\Big(\sum_y y\,p_Y(y)\Big) = E[X]E[Y],

so the covariance is zero.

Exercise 8 · Variance of a sum

Prove $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X, Y)$ .

Show solution

Let $\tilde X = X - \mu_X$ and $\tilde Y = Y - \mu_Y$ . Then

\operatorname{Var}(X + Y) = E[(\tilde X + \tilde Y)^2] = E[\tilde X^2] + E[\tilde Y^2] + 2E[\tilde X\tilde Y].

Exercise 9 · Correlation is bounded

Prove $|\rho_{XY}| \le 1$ . Hint: consider $\operatorname{Var}(Y - tX) \ge 0$ for every $t$ .

Show solution

Step 1. $h(t) = \operatorname{Var}(Y - tX) = \sigma_Y^2 - 2t\,\sigma_{XY} + t^2\sigma_X^2 \ge 0$ for all $t$ .

Step 2. A quadratic in $t$ that never goes negative has a non-positive discriminant:

4\sigma_{XY}^2 - 4\sigma_X^2\sigma_Y^2 \le 0 \;\Rightarrow\; \sigma_{XY}^2 \le \sigma_X^2\sigma_Y^2 \;\Rightarrow\; |\rho| \le 1.

Step 3. Equality holds exactly when $Y - tX$ is constant for some $t$ , i.e. when $Y$ is a linear function of $X$ .

Exercise 10 · The slope on a binary regressor

Let $X \in \{0, 1\}$ with $P(X = 1) = p$ . Prove

\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)} = E[Y \mid X = 1] - E[Y \mid X = 0].

Show solution

Step 1. Let $m_1 = E[Y \mid X = 1]$ and $m_0 = E[Y \mid X = 0]$ .

Step 2. Then $E[XY] = p\,m_1$ , $E[X] = p$ and $E[Y] = p\,m_1 + (1 - p)m_0$ .

Step 3. So

\operatorname{Cov}(X, Y) = p\,m_1 - p\big(p\,m_1 + (1 - p)m_0\big) = p(1 - p)(m_1 - m_0).

Step 4. Dividing by $\operatorname{Var}(X) = p(1 - p)$ gives $m_1 - m_0$ .

Step 5. For the table above: $0.06/0.24 = 0.25 = 0.75 - 0.50$ .

Step 6. Regressing an outcome on a treatment dummy is a difference in means.

Part 3 · Applied

Exercise 11 · A U-shaped health risk

Daily temperature deviation from the seasonal norm, $X$ , is $-1$ (cold snap), 0 or $+1$ (heat wave), each with probability 1/3. Excess hospital admissions are $Y = X^2$ (both extremes raise admissions). Show that $\operatorname{Cov}(X, Y) = 0$ although $Y$ is completely determined by $X$ .

Show solution

Step 1. $E[X] = (-1 + 0 + 1)/3 = 0$ .

Step 2. $XY = X^3$ , so $E[XY] = ((-1)^3 + 0 + 1^3)/3 = 0$ .

Step 3. $\operatorname{Cov}(X, Y) = E[XY] - E[X]E[Y] = 0 - 0 \cdot E[Y] = 0$ .

Answer: the covariance (and correlation) is zero, yet temperature determines admissions exactly. Covariance only measures linear association; a linear regression of admissions on temperature would wrongly suggest no relationship.

Exercise 12 · Diversifying programmes

A foundation splits its budget equally between programmes A and B with impact standard deviations 10 and 20 and correlation $-0.5$ . Find the standard deviation of total impact $0.5A + 0.5B$ .

Show solution

Step 1. Covariance of the two programmes: $\operatorname{Cov}(A, B) = \rho\sigma_A\sigma_B = -0.5 \times 10 \times 20 = -100$ .

Step 2. Variance of the half-and-half portfolio:

\operatorname{Var}(0.5A + 0.5B) = 0.25(100) + 0.25(400) + 2(0.5)(0.5)(-100) = 25 + 100 - 50 = 75.

Step 3. sd $= \sqrt{75} \approx 8.66$ .

Answer: 8.66, below the sd of either programme alone (10 and 20): negatively correlated programmes hedge each other.

Exercise 13 · Omitted ability in a wage regression

Log wages depend on schooling $X$ (years, standardised so $\operatorname{Var}(X) = 1$ ) and ability $U$ : $Y = 2X + 3U + \varepsilon$ , with $\varepsilon$ uncorrelated with $X$ and $U$ , and $\operatorname{Cov}(X, U) = 0.5$ (abler people stay in school longer). What slope do you get from regressing $Y$ on $X$ alone?

Show solution

Step 1. The short-regression slope is $\operatorname{Cov}(X, Y)/\operatorname{Var}(X)$ .

Step 2. Expand the covariance using the model:

\operatorname{Cov}(X, Y) = 2\operatorname{Var}(X) + 3\operatorname{Cov}(X, U) + \operatorname{Cov}(X, \varepsilon) = 2(1) + 3(0.5) + 0 = 3.5.

Step 3. Divide by $\operatorname{Var}(X) = 1$ : slope $= 3.5$ .

Step 4. Bias $= 3.5 - 2 = 1.5$ = (effect of ability on wages, 3) × (slope of ability on schooling, 0.5).

Answer: 3.5. Leaving out ability overstates the return to schooling by 75%.

Exercise 14 · When is a training coefficient causal?

Earnings follow $Y = \beta_0 + \beta_1X + \varepsilon$ , where $X$ is hours of job training and $\varepsilon$ collects every other cause of earnings (motivation, local labour demand, …). Show that the population regression slope $\operatorname{Cov}(X, Y)/\operatorname{Var}(X)$ equals $\beta_1$ if and only if $\operatorname{Cov}(X, \varepsilon) = 0$ .

Show solution

Step 1. Take the covariance of both sides with $X$ (constants have zero covariance):

\operatorname{Cov}(X, Y) = \beta_1\operatorname{Var}(X) + \operatorname{Cov}(X, \varepsilon).

Step 2. Divide by $\operatorname{Var}(X)$ :

\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)} = \beta_1 + \frac{\operatorname{Cov}(X, \varepsilon)}{\operatorname{Var}(X)}.

Step 3. The second term vanishes exactly when $\operatorname{Cov}(X, \varepsilon) = 0$ .

Answer: the slope is causal only if training hours are uncorrelated with all other causes of earnings. If motivated workers take more training, the slope overstates $\beta_1$ . This exogeneity condition is what every identification strategy in Phase 5 tries to secure.

Exercise 15 · Prediction with a bivariate normal

Years of education $X$ have mean 12 and sd 2; log wages $Y$ have mean 3 and sd 0.5; $\rho = 0.4$ , jointly normal. Find $E[Y \mid X = 16]$ and $\operatorname{sd}(Y \mid X)$ .

Show solution

Step 1. Conditional mean of a bivariate normal:

E[Y \mid X = 16] = \mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(16 - \mu_X) = 3 + 0.4 \times \frac{0.5}{2} \times 4 = 3 + 0.4 = 3.4.

Step 2. Conditional variance: $\operatorname{Var}(Y \mid X) = \sigma_Y^2(1 - \rho^2) = 0.25 \times 0.84 = 0.21$ .

Step 3. Conditional sd: $\sqrt{0.21} \approx 0.458$ , versus the unconditional 0.5.

Answer: predicted log wage 3.4 for 16 years of schooling, with sd 0.458. A correlation of 0.4 removes only 16% of the variance; and this slope is a predictive relationship, not a causal return to schooling.

Verify in Python

import numpy as np

joint = np.array([[0.30, 0.30],    # rows: X = 0, 1; columns: Y = 0, 1
                  [0.10, 0.30]])
x = np.array([0, 1]); y = np.array([0, 1])
px, py = joint.sum(1), joint.sum(0)
exy = sum(joint[i, j] * x[i] * y[j] for i in range(2) for j in range(2))
cov = exy - (px @ x) * (py @ y)
vx, vy = px @ x**2 - (px @ x)**2, py @ y**2 - (py @ y)**2
print(f"Cov = {cov:.2f}, corr = {cov / np.sqrt(vx * vy):.2f}, slope = {cov / vx:.2f}")

# Omitted-variable bias by simulation (Exercise 13)
rng = np.random.default_rng(7)
n = 500_000
X = rng.normal(0, 1, n)
U = 0.5 * X + rng.normal(0, 1, n)          # Cov(X, U) = 0.5
Y = 2 * X + 3 * U + rng.normal(0, 1, n)
print("short-regression slope:", round(np.cov(X, Y)[0, 1] / X.var(ddof=1), 3))

Key takeaways

Covariance measures linear co-movement; correlation rescales it to $[-1, 1]$ .
A regression slope is a covariance divided by a variance; on a dummy it is a difference in means.
Leaving out a variable correlated with $X$ biases the slope by (its effect) × (its slope on $X$ ).

Day 5. Joint Distributions, Covariance & Correlation

Theory

Joint, marginal and conditional

Covariance and correlation

Covariance is the regression slope's numerator

The bivariate normal

Part 1 · Warm-up

Exercise 1 · Marginals

Exercise 2 · The product moment

Exercise 3 · Covariance

Exercise 4 · Correlation

Exercise 5 · Conditionals and independence

Part 2 · Proofs

Exercise 6 · Covariance shortcut

Exercise 7 · Independence implies zero covariance

Exercise 8 · Variance of a sum

Exercise 9 · Correlation is bounded

Exercise 10 · The slope on a binary regressor

Part 3 · Applied

Exercise 11 · A U-shaped health risk

Exercise 12 · Diversifying programmes

Exercise 13 · Omitted ability in a wage regression

Exercise 14 · When is a training coefficient causal?

Exercise 15 · Prediction with a bivariate normal

Verify in Python

Key takeaways

Simulation Lab: Bivariate Normal in 3D