Day 21: Eigenvalues, Eigenvectors & Quadratic Forms

Theory

Definition

A non-zero vector $v$ is an eigenvector of a square matrix $A$ , with eigenvalue $\lambda$ , if

Av = \lambda v.

$A$ maps $v$ onto its own line, stretching it by $\lambda$ . Eigenvalues solve the characteristic equation $\det(A - \lambda I) = 0$ . For 2×2 matrices,

\lambda^2 - \operatorname{tr}(A)\,\lambda + \det(A) = 0, \qquad \lambda_1 + \lambda_2 = \operatorname{tr}A, \qquad \lambda_1\lambda_2 = \det A.

Symmetric matrices: the spectral theorem

If $A = A^{\top}$ (covariance matrices, $X^{\top}X$ ), then all eigenvalues are real, eigenvectors can be chosen orthonormal, and

A = Q\Lambda Q^{\top} = \sum_j \lambda_j q_jq_j^{\top}, \qquad Q^{\top}Q = I.

Consequences: $A^m = Q\Lambda^mQ^{\top}$ , $A^{-1} = Q\Lambda^{-1}Q^{\top}$ (if all $\lambda_j \ne 0$ ), and $A^{1/2} = Q\Lambda^{1/2}Q^{\top}$ .

Quadratic forms and definiteness

A quadratic form is $x^{\top}Ax$ . Writing $x$ in eigen-coordinates $z = Q^{\top}x$ gives $x^{\top}Ax = \sum_j\lambda_jz_j^2$ , so a symmetric $A$ is

	eigenvalues	$x^{\top}Ax$ for $x \ne 0$
positive definite (PD)	all $> 0$	always $> 0$
positive semi-definite (PSD)	all $\ge 0$	always $\ge 0$
indefinite	mixed signs	both signs

The Rayleigh quotient $x^{\top}Ax/x^{\top}x$ ranges between $\lambda_{\min}$ and $\lambda_{\max}$ .

Three uses in econometrics

Principal components: the eigenvectors of a covariance matrix are the directions of greatest variance; eigenvalues are those variances.
Collinearity: the condition number $\lambda_{\max}/\lambda_{\min}$ of $X^{\top}X$ (for standardised regressors) measures how close $X$ is to rank deficiency.
Dynamics: $x_{t+1} = Ax_t$ converges to zero iff every $|\lambda_j| < 1$ .

Part 1 · Warm-up

Exercise 1 · A diagonal matrix

Find the eigenvalues and eigenvectors of $\begin{pmatrix}2 & 0\\0 & 5\end{pmatrix}$ .

Show solution

$\lambda = 2$ with $v = (1, 0)$ and $\lambda = 5$ with $v = (0, 1)$ : a diagonal matrix stretches along the axes.

Exercise 2 · A symmetric matrix

Find the eigenvalues and eigenvectors of $A = \begin{pmatrix}2 & 1\\1 & 2\end{pmatrix}$ .

Show solution

Step 1. Characteristic equation: $\lambda^2 - \operatorname{tr}(A)\lambda + \det A = \lambda^2 - 4\lambda + 3 = (\lambda - 3)(\lambda - 1) = 0$ , so $\lambda = 3, 1$ .

Step 2. For $\lambda = 3$ : $(A - 3I)v = \begin{pmatrix}-1 & 1\\1 & -1\end{pmatrix}v = 0 \Rightarrow v_1 = v_2$ , e.g. $v = (1, 1)$ .

Step 3. For $\lambda = 1$ : $(A - I)v = \begin{pmatrix}1 & 1\\1 & 1\end{pmatrix}v = 0 \Rightarrow v_1 = -v_2$ , e.g. $v = (1, -1)$ .

Step 4. Check orthogonality: $(1, 1)\cdot(1, -1) = 0$ .

Answer: $\lambda = 3$ along $(1, 1)$ and $\lambda = 1$ along $(1, -1)$ .

Exercise 3 · Trace and determinant

Check the trace and determinant identities for Exercise 2.

Show solution

$\operatorname{tr} = 4 = 3 + 1$ and $\det = 3 = 3 \times 1$ . ✓

Exercise 4 · Evaluating a quadratic form

Compute $x^{\top}Ax$ for Exercise 2's $A$ with $x = (1, -1)$ and with $x = (1, 1)$ .

Show solution

Step 1. $x^{\top}Ax = 2x_1^2 + 2x_1x_2 + 2x_2^2$ .

Step 2. At $(1, -1)$ : $2 - 2 + 2 = 2 = 1\cdot\|x\|^2$ .

Step 3. At $(1, 1)$ : $6 = 3\cdot\|x\|^2$ .

Step 4. Along eigenvectors, the form is $\lambda\|x\|^2$ .

Exercise 5 · Classifying a form

Write $3x^2 + 4xy + y^2$ as $x^{\top}Ax$ and classify it.

Show solution

Step 1. $A = \begin{pmatrix}3 & 2\\2 & 1\end{pmatrix}$ (split the cross term evenly).

Step 2. $\det A = 3 - 4 = -1 < 0$ , so the eigenvalues have opposite signs: indefinite (e.g. $x = (1, -2)$ gives $3 - 8 + 4 = -1$ ).

Part 2 · Proofs

Exercise 6 · Orthogonal eigenvectors

Prove that eigenvectors of a symmetric matrix belonging to distinct eigenvalues are orthogonal.

Show solution

Step 1. Let $Av_1 = \lambda_1v_1$ , $Av_2 = \lambda_2v_2$ .

Step 2. Then $\lambda_1v_2^{\top}v_1 = v_2^{\top}Av_1 = (Av_2)^{\top}v_1 = \lambda_2v_2^{\top}v_1$ , using $A = A^{\top}$ .

Step 3. So $(\lambda_1 - \lambda_2)v_2^{\top}v_1 = 0$ , and $\lambda_1 \ne \lambda_2$ forces $v_2^{\top}v_1 = 0$ .

Exercise 7 · Trace and determinant from the roots

For a 2×2 matrix, show $\lambda_1 + \lambda_2 = \operatorname{tr}A$ and $\lambda_1\lambda_2 = \det A$ .

Show solution

Step 1. $\det(A - \lambda I) = (a - \lambda)(d - \lambda) - bc = \lambda^2 - (a + d)\lambda + (ad - bc)$ .

Step 2. A monic quadratic with roots $\lambda_1, \lambda_2$ equals $\lambda^2 - (\lambda_1 + \lambda_2)\lambda + \lambda_1\lambda_2$ .

Step 3. Match coefficients.

Exercise 8 · Powers and inverses

Show that if $Av = \lambda v$ then $A^mv = \lambda^mv$ , and, if $A$ is invertible, $A^{-1}v = \lambda^{-1}v$ .

Show solution

$A^2v = A(\lambda v) = \lambda^2v$ , and by induction $A^mv = \lambda^mv$ . Applying $A^{-1}$ to $Av = \lambda v$ gives $v = \lambda A^{-1}v$ ; invertibility forces $\lambda \ne 0$ , so $A^{-1}v = \lambda^{-1}v$ .

Exercise 9 · Positive definite ⇔ positive eigenvalues

Using $A = Q\Lambda Q^{\top}$ , prove that a symmetric matrix is PD iff all its eigenvalues are positive.

Show solution

Step 1. With $z = Q^{\top}x$ (and $z \ne 0 \iff x \ne 0$ since $Q$ is invertible), $x^{\top}Ax = z^{\top}\Lambda z = \sum_j\lambda_jz_j^2$ .

Step 2. If every $\lambda_j > 0$ this is positive for $z \ne 0$ .

Step 3. Conversely, taking $x = q_j$ gives $x^{\top}Ax = \lambda_j$ , which must be positive.

Exercise 10 · The Rayleigh quotient

Prove that $\max_{\|x\| = 1}x^{\top}Ax = \lambda_{\max}$ for symmetric $A$ .

Show solution

Step 1. $\|x\| = 1 \iff \|z\| = 1$ (since $Q$ preserves length).

Step 2. Then $x^{\top}Ax = \sum_j\lambda_jz_j^2 \le \lambda_{\max}\sum_jz_j^2 = \lambda_{\max}$ , with equality at $x = q_{\max}$ .

Step 3. This is the first principal component: the unit direction of greatest variance.

Part 3 · Applied

Exercise 11 · Principal components

Two test scores have covariance matrix $\Sigma = \begin{pmatrix}4 & 2\\2 & 3\end{pmatrix}$ . Find the eigenvalues and the share of total variance captured by the first principal component.

Show solution

Step 1. $\operatorname{tr}\Sigma = 7$ and $\det\Sigma = 4(3) - 2^2 = 8$ , so $\lambda^2 - 7\lambda + 8 = 0$ .

Step 2. $\lambda = \frac{7 \pm \sqrt{49 - 32}}{2} = \frac{7 \pm \sqrt{17}}{2} \approx 5.56,\; 1.44$ .

Step 3. Total variance is the trace, 7, so the first component's share is $5.56/7 \approx 79.5\%$ .

Answer: one "ability" index (the first principal component) summarises about 80% of the variation in the two scores.

Exercise 12 · Stability of a dynamic system

Employment shares evolve as $x_{t+1} = Ax_t$ with $A = \begin{pmatrix}0.5 & 0.2\\0.1 & 0.6\end{pmatrix}$ (deviations from steady state). Do deviations die out?

Show solution

Step 1. $\operatorname{tr}A = 0.5 + 0.6 = 1.1$ and $\det A = 0.5(0.6) - 0.2(0.1) = 0.28$ .

Step 2. $\lambda^2 - 1.1\lambda + 0.28 = 0 \Rightarrow \lambda = \frac{1.1 \pm \sqrt{1.21 - 1.12}}{2} = \frac{1.1 \pm 0.3}{2} = 0.7,\; 0.4$ .

Step 3. Both satisfy $|\lambda| < 1$ , so $A^t \to 0$ and $x_t = A^tx_0 \to 0$ .

Answer: yes; shocks die out, by 30% per period along the slower direction ( $\lambda = 0.7$ ).

Exercise 13 · Two almost identical wealth measures

A poverty study includes both an asset index and household income as regressors. After standardising, they have correlation 0.99, so $X^{\top}X/n = \begin{pmatrix}1 & 0.99\\0.99 & 1\end{pmatrix}$ . Find its eigenvalues and condition number, and say which combination of the two coefficients the data cannot pin down.

Show solution

Step 1. For $\begin{pmatrix}1 & \rho\\\rho & 1\end{pmatrix}$ the eigenvectors are $(1, 1)$ and $(1, -1)$ with eigenvalues $1 + \rho$ and $1 - \rho$ (as in Exercise 2).

Step 2. With $\rho = 0.99$ : eigenvalues 1.99 and 0.01.

Step 3. Condition number $= 1.99/0.01 = 199$ .

Step 4. The tiny eigenvalue belongs to $(1, -1)$ : the data contain almost no variation in "assets minus income".

Answer: condition number 199. The sum of the two coefficients is well estimated, their difference hardly at all (see the collinearity simulator).

Exercise 14 · Ridge repairs the spectrum

Add $\lambda = 0.1$ to the diagonal (ridge regression). What happens to the eigenvalues and condition number?

Show solution

Step 1. Adding $\lambda I$ keeps the eigenvectors and adds $\lambda$ to each eigenvalue: $(X^{\top}X/n + 0.1I)q_j = (\lambda_j + 0.1)q_j$ .

Step 2. New eigenvalues: $1.99 + 0.1 = 2.09$ and $0.01 + 0.1 = 0.11$ .

Step 3. New condition number: $2.09/0.11 = 19$ .

Answer: ridge cuts the condition number from 199 to 19, trading a little bias for a large drop in variance (Day 8).

Exercise 15 · How different are two pupils?

A matching estimator compares a tutored pupil with an untutored one. Their two baseline test scores (maths, reading) differ by $d = (2, 1)$ points, and the scores' covariance matrix is $\Sigma$ from Exercise 11. Compute the Mahalanobis distance $\sqrt{d^{\top}\Sigma^{-1}d}$ and compare it with the Euclidean distance.

Show solution

Step 1. $\det\Sigma = 8$ , so $\Sigma^{-1} = \frac18\begin{pmatrix}3 & -2\\-2 & 4\end{pmatrix}$ .

Step 2. Quadratic form: $d^{\top}\Sigma^{-1}d = \frac18\big(3(2)^2 + 2(-2)(2)(1) + 4(1)^2\big) = \frac18(12 - 8 + 4) = 1$ .

Step 3. Mahalanobis distance $= \sqrt1 = 1$ ; Euclidean distance $= \sqrt{4 + 1} \approx 2.24$ .

Answer: 1 versus 2.24. The difference lies along the high-variance "general ability" direction, where such gaps are common, so the two pupils are a good match. Mahalanobis distance is scale-free and accounts for correlation.

Verify in Python

import numpy as np

A = np.array([[2, 1], [1, 2.]])
lam, Q = np.linalg.eigh(A)
print("eigenvalues:", lam, "\neigenvectors (columns):\n", Q.round(3))
print("Q diag(lam) Q' == A:", np.allclose(Q @ np.diag(lam) @ Q.T, A))

S = np.array([[4, 2], [2, 3.]])
ev = np.linalg.eigvalsh(S)[::-1]
print("PC variances:", ev.round(3), " first PC share:", round(ev[0] / ev.sum(), 3))

D = np.array([[0.5, 0.2], [0.1, 0.6]])
print("dynamic eigenvalues:", np.sort(np.linalg.eigvals(D)).round(3))

C = np.array([[1, 0.99], [0.99, 1]])
print("condition number:", round(np.linalg.cond(C)), " with ridge 0.1:", round(np.linalg.cond(C + 0.1 * np.eye(2))))
d = np.array([2, 1.])
print("Mahalanobis:", np.sqrt(d @ np.linalg.solve(S, d)), " Euclidean:", round(np.linalg.norm(d), 3))

Key takeaways

Eigenvectors are the directions a matrix only stretches; eigenvalues are the stretch factors.
Symmetric matrices diagonalise in an orthonormal basis, so definiteness is read off the eigenvalues.
Eigenvalues tell you the main directions of variance, how collinear your regressors are, and whether a dynamic system is stable.

Day 21. Eigenvalues, Eigenvectors & Quadratic Forms

Theory

Definition

Symmetric matrices: the spectral theorem

Quadratic forms and definiteness

Three uses in econometrics

Part 1 · Warm-up

Exercise 1 · A diagonal matrix

Exercise 2 · A symmetric matrix

Exercise 3 · Trace and determinant

Exercise 4 · Evaluating a quadratic form

Exercise 5 · Classifying a form

Part 2 · Proofs

Exercise 6 · Orthogonal eigenvectors

Exercise 7 · Trace and determinant from the roots

Exercise 8 · Powers and inverses

Exercise 9 · Positive definite ⇔ positive eigenvalues

Exercise 10 · The Rayleigh quotient

Part 3 · Applied

Exercise 11 · Principal components

Exercise 12 · Stability of a dynamic system

Exercise 13 · Two almost identical wealth measures

Exercise 14 · Ridge repairs the spectrum

Exercise 15 · How different are two pupils?

Verify in Python

Key takeaways

Simulation Lab: Eigenvectors & Quadratic Forms