Day 22: Positive-Definite Matrices & Variance–Covariance Matrices

Theory

Random vectors

For a random vector $X = (X_1, \dots, X_k)^{\top}$ , the mean vector is $\mu = E[X]$ and the variance–covariance matrix is

\Sigma = \operatorname{Var}(X) = E\big[(X - \mu)(X - \mu)^{\top}\big], \qquad \Sigma_{jj} = \operatorname{Var}(X_j), \quad \Sigma_{j\ell} = \operatorname{Cov}(X_j, X_\ell).

$\Sigma$ is symmetric. Its correlation matrix is $D^{-1/2}\Sigma D^{-1/2}$ with $D = \operatorname{diag}(\Sigma)$ .

The two rules you will use every week

\operatorname{Var}(a^{\top}X) = a^{\top}\Sigma a, \qquad \operatorname{Var}(AX + b) = A\Sigma A^{\top}.

They give the variance of a portfolio, a contrast between treatment arms, a cumulative event-study effect, or a predicted value $x_0^{\top}\hat\beta$ : just plug in $\operatorname{Var}(\hat\beta)$ .

Positive (semi-)definiteness

Because $a^{\top}\Sigma a$ is a variance, it can never be negative: every covariance matrix is PSD. It is PD unless some combination of the variables is constant (e.g. shares that sum to one). A symmetric matrix that is not PSD (e.g. a "correlation matrix" assembled pairwise from different samples) cannot be the covariance matrix of anything.

Cholesky factor and simulation

A PD matrix factors as $\Sigma = LL^{\top}$ with $L$ lower-triangular (the Cholesky factor). If $Z \sim N(0, I)$ , then

X = \mu + LZ \sim N(\mu, \Sigma).

This is how simulations generate correlated covariates, errors or potential outcomes.

Sample covariance

With the centred data matrix $\tilde X$ ( $n \times k$ ), $S = \frac{1}{n - 1}\tilde X^{\top}\tilde X$ , which is PSD by Day 17.

Part 1 · Warm-up

Let $\Sigma = \begin{pmatrix}4 & 2\\2 & 9\end{pmatrix}$ .

Exercise 1 · Variance of a sum

Find $\operatorname{Var}(X_1 + X_2)$ .

Show solution

$a = (1, 1)$ : $a^{\top}\Sigma a = 4 + 9 + 2(2) = 17$ .

Exercise 2 · Variance of a difference

Find $\operatorname{Var}(X_1 - X_2)$ .

Show solution

$a = (1, -1)$ : $4 + 9 - 2(2) = 9$ .

Exercise 3 · Correlation

Find the correlation between $X_1$ and $X_2$ .

Show solution

$2/(\sqrt4\sqrt9) = 2/6 \approx 0.333$ .

Exercise 4 · A valid covariance matrix?

Can $\begin{pmatrix}1 & 2\\2 & 1\end{pmatrix}$ be a covariance matrix?

Show solution

Step 1. No.

Step 2. Its determinant is $-3 < 0$ , so it has a negative eigenvalue; equivalently the implied correlation is 2.

Step 3. For $a = (1, -1)$ , $a^{\top}\Sigma a = 1 + 1 - 4 = -2$ : a negative variance.

Exercise 5 · Cholesky by hand

Find the Cholesky factor of $\Sigma$ .

Show solution

Step 1. Write $L = \begin{pmatrix}\ell_{11} & 0\\\ell_{21} & \ell_{22}\end{pmatrix}$ , so $LL^{\top} = \begin{pmatrix}\ell_{11}^2 & \ell_{11}\ell_{21}\\\ell_{11}\ell_{21} & \ell_{21}^2 + \ell_{22}^2\end{pmatrix}$ .

Step 2. Match to $\Sigma$ : $\ell_{11}^2 = 4 \Rightarrow \ell_{11} = 2$ .

Step 3. $\ell_{11}\ell_{21} = 2 \Rightarrow \ell_{21} = 1$ .

Step 4. $\ell_{21}^2 + \ell_{22}^2 = 9 \Rightarrow \ell_{22} = \sqrt8 \approx 2.828$ .

Answer: $L = \begin{pmatrix}2 & 0\\1 & 2.828\end{pmatrix}$ .

Part 2 · Proofs

Exercise 6 · Variance of a linear combination

Prove $\operatorname{Var}(a^{\top}X) = a^{\top}\Sigma a$ .

Show solution

$a^{\top}X - E[a^{\top}X] = a^{\top}(X - \mu)$ , a scalar equal to its own transpose. So

\operatorname{Var}(a^{\top}X) = E\big[a^{\top}(X - \mu)(X - \mu)^{\top}a\big] = a^{\top}E\big[(X - \mu)(X - \mu)^{\top}\big]a = a^{\top}\Sigma a.

Exercise 7 · Linear transformations

Prove $\operatorname{Var}(AX + b) = A\Sigma A^{\top}$ .

Show solution

$AX + b - E[AX + b] = A(X - \mu)$ , so $\operatorname{Var} = E[A(X - \mu)(X - \mu)^{\top}A^{\top}] = A\Sigma A^{\top}$ .

Exercise 8 · Covariance matrices are PSD

Prove that every covariance matrix is positive semi-definite.

Show solution

For any $a$ , $a^{\top}\Sigma a = \operatorname{Var}(a^{\top}X) \ge 0$ (Exercise 6).

Exercise 9 · Simulating with Cholesky

Prove that if $Z \sim N(0, I)$ then $X = \mu + LZ$ has mean $\mu$ and covariance $LL^{\top} = \Sigma$ .

Show solution

$E[X] = \mu + LE[Z] = \mu$ , and by Exercise 7, $\operatorname{Var}(X) = L\,I\,L^{\top} = \Sigma$ . (A linear function of normals is normal, so $X \sim N(\mu, \Sigma)$ .)

Exercise 10 · What PD implies about entries

Show that a PD matrix has positive diagonal entries and $|\sigma_{12}| < \sqrt{\sigma_{11}\sigma_{22}}$ .

Show solution

Take $a = e_j$ (the $j$ th unit vector): $\sigma_{jj} = e_j^{\top}\Sigma e_j > 0$ . The 2×2 leading block is also PD, so its determinant (product of its eigenvalues) is positive: $\sigma_{11}\sigma_{22} - \sigma_{12}^2 > 0$ .

Part 3 · Applied

Exercise 11 · Portfolio risk

Two assets have return sds 10% and 20% and correlation 0.3. With weights $w = (0.6, 0.4)$ , find the portfolio sd.

Show solution

Step 1. Covariance: $0.3 \times 10 \times 20 = 60$ , so $\Sigma = \begin{pmatrix}100 & 60\\60 & 400\end{pmatrix}$ .

Step 2. $w^{\top}\Sigma w = 0.6^2(100) + 0.4^2(400) + 2(0.6)(0.4)(60) = 36 + 64 + 28.8 = 128.8$ .

Step 3. sd $= \sqrt{128.8} \approx 11.35$ .

Answer: portfolio sd ≈ 11.35%.

Exercise 12 · Minimum-variance weights

For the same assets, find the weight $w_1$ that minimises portfolio variance (with $w_2 = 1 - w_1$ ).

Show solution

Step 1. Variance as a function of $w_1$ : $V(w_1) = w_1^2\sigma_1^2 + (1 - w_1)^2\sigma_2^2 + 2w_1(1 - w_1)\sigma_{12}$ .

Step 2. Set $V'(w_1) = 2w_1\sigma_1^2 - 2(1 - w_1)\sigma_2^2 + 2(1 - 2w_1)\sigma_{12} = 0$ and solve:

w_1 = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}}.

Step 3. Plug in: $w_1 = (400 - 60)/(100 + 400 - 120) = 340/380 \approx 0.895$ .

Answer: about 89.5% in the less risky asset (Day 26 derives the general version with a Lagrange multiplier).

Exercise 13 · Are two effects different?

A regression gives $\hat\beta_1 = 0.5$ and $\hat\beta_2 = 0.2$ with $\operatorname{Var}(\hat\beta_1) = 0.04$ , $\operatorname{Var}(\hat\beta_2) = 0.09$ , $\operatorname{Cov} = 0.02$ . Test $\beta_1 = \beta_2$ .

Show solution

Step 1. Contrast $a = (1, -1)$ : $\operatorname{Var}(\hat\beta_1 - \hat\beta_2) = 0.04 + 0.09 - 2(0.02) = 0.09$ .

Step 2. SE $= 0.3$ ; estimate $= 0.5 - 0.2 = 0.3$ .

Step 3. $z = 0.3/0.3 = 1$ , $p \approx 0.32$ .

Answer: not significantly different. Ignoring the covariance would give SE $\sqrt{0.13} \approx 0.36$ instead.

Exercise 14 · A cumulative effect

An event study estimates three post-period effects, each with variance 0.01 and pairwise covariance 0.005. Find the SE of their sum.

Show solution

Step 1. The sum is $\mathbf 1^{\top}\hat\beta$ , so its variance is $\mathbf 1^{\top}V\mathbf 1$ = the sum of all entries of $V$ .

Step 2. Diagonal: $3 \times 0.01 = 0.03$ . Off-diagonal: 6 entries × 0.005 = 0.03.

Step 3. Total 0.06, SE $= \sqrt{0.06} \approx 0.245$ .

Answer: 0.245; assuming independence would give 0.173, far too small.

Exercise 15 · An impossible set of survey correlations

A report on a household survey lists pairwise correlations computed from different sub-samples: income–education 0.9, income–health 0.9, and education–health −0.9. Could these be the correlations of three real variables?

Show solution

Step 1. A valid correlation matrix must be PSD, so its determinant (the product of the eigenvalues) cannot be negative.

Step 2. Cofactor expansion of $R = \begin{pmatrix}1 & 0.9 & 0.9\\0.9 & 1 & -0.9\\0.9 & -0.9 & 1\end{pmatrix}$ :

\det R = 1(1 - 0.81) - 0.9(0.9 + 0.81) + 0.9(-0.81 - 0.9) = 0.19 - 1.539 - 1.539 \approx -2.89.

Step 3. Negative, so $R$ has a negative eigenvalue (−0.8, from the code).

Answer: impossible. If income is strongly correlated with both education and health, those two cannot be strongly negatively correlated. Pairwise estimates from different samples can produce such incoherent matrices.

Verify in Python

import numpy as np

S = np.array([[4, 2], [2, 9.]])
for a in ([1, 1], [1, -1]):
    a = np.array(a); print("Var(a'X) for a =", a, ":", a @ S @ a)
L = np.linalg.cholesky(S); print("Cholesky L =\n", L.round(3))

rng = np.random.default_rng(22)
Z = rng.normal(size=(2, 200_000)); X = (L @ Z).T
print("simulated covariance:\n", np.cov(X.T).round(2))

w = np.array([0.6, 0.4]); Sig = np.array([[100, 60], [60, 400.]])
print("portfolio sd:", round(np.sqrt(w @ Sig @ w), 2))
V = np.array([[0.01, 0.005, 0.005], [0.005, 0.01, 0.005], [0.005, 0.005, 0.01]])
print("SE of cumulative effect:", round(np.sqrt(np.ones(3) @ V @ np.ones(3)), 3))
R = np.array([[1, 0.9, 0.9], [0.9, 1, -0.9], [0.9, -0.9, 1]])
print("eigenvalues of the 'correlation matrix':", np.linalg.eigvalsh(R).round(3))

Key takeaways

$\operatorname{Var}(a^{\top}X) = a^{\top}\Sigma a$ and $\operatorname{Var}(AX + b) = A\Sigma A^{\top}$ give the variance of any linear combination.
Covariance matrices are PSD; a matrix that is not PSD cannot describe real data.
Cholesky factors turn independent normals into correlated ones, the basis of most simulation studies.

Day 22. Positive-Definite Matrices & Variance–Covariance Matrices

Theory

Random vectors

The two rules you will use every week

Positive (semi-)definiteness

Cholesky factor and simulation

Sample covariance

Part 1 · Warm-up

Exercise 1 · Variance of a sum

Exercise 2 · Variance of a difference

Exercise 3 · Correlation

Exercise 4 · A valid covariance matrix?

Exercise 5 · Cholesky by hand

Part 2 · Proofs

Exercise 6 · Variance of a linear combination

Exercise 7 · Linear transformations

Exercise 8 · Covariance matrices are PSD

Exercise 9 · Simulating with Cholesky

Exercise 10 · What PD implies about entries

Part 3 · Applied

Exercise 11 · Portfolio risk

Exercise 12 · Minimum-variance weights

Exercise 13 · Are two effects different?

Exercise 14 · A cumulative effect

Exercise 15 · An impossible set of survey correlations

Verify in Python

Key takeaways

Simulation Lab: Covariance Ellipses & a′Σa