Day 24: Matrix Calculus & Gradients · Math for Causal Inference

Theory

Layout

For a scalar function $f(x)$ of a vector $x \in \mathbb{R}^k$ , the gradient is the column vector $\partial f/\partial x = (\partial f/\partial x_1, \dots, \partial f/\partial x_k)^{\top}$ , and the Hessian is the $k \times k$ matrix $\partial^2 f/\partial x\,\partial x^{\top}$ . For a vector function $g(x) \in \mathbb{R}^m$ , the Jacobian $\partial g/\partial x^{\top}$ is $m \times k$ .

Four rules cover almost everything

Function	Gradient
$a^{\top}x$	$a$
$x^{\top}Ax$	$(A + A^{\top})x$ , which is $2Ax$ if $A$ is symmetric
$Ax$ (vector)	Jacobian $A$
$f(g(x))$	chain rule: $\frac{\partial g}{\partial x^{\top}}^{\!\top}\nabla f(g(x))$

The least-squares objective

S(\beta) = (y - X\beta)^{\top}(y - X\beta) = y^{\top}y - 2\beta^{\top}X^{\top}y + \beta^{\top}X^{\top}X\beta,

\nabla S(\beta) = -2X^{\top}y + 2X^{\top}X\beta = -2X^{\top}(y - X\beta), \qquad \nabla^2 S = 2X^{\top}X.

Setting the gradient to zero gives the normal equations $X^{\top}X\hat\beta = X^{\top}y$ . The Hessian is PSD (Day 17), so $S$ is convex and the solution is a minimum (Day 25).

The same template, other estimators

Ridge: $S(\beta) + \lambda\beta^{\top}\beta \Rightarrow \hat\beta = (X^{\top}X + \lambda I)^{-1}X^{\top}y$ .
Weighted least squares: $(y - X\beta)^{\top}W(y - X\beta) \Rightarrow \hat\beta = (X^{\top}WX)^{-1}X^{\top}Wy$ .
Logit: $\nabla\ell = X^{\top}(y - p)$ and $\nabla^2\ell = -X^{\top}VX$ with $V = \operatorname{diag}(p_i(1 - p_i))$ .

Causal link. Weighted least squares with weights $1/\hat e(x)$ and $1/(1 - \hat e(x))$ is how IPW regressions are run; ridge and lasso are the nuisance learners in double machine learning.

Part 1 · Warm-up

Exercise 1 · A linear function

Find the gradient of $f(x) = a^{\top}x$ with $a = (2, -1, 3)$ .

Show solution

$\nabla f = a = (2, -1, 3)^{\top}$ , the same at every $x$ .

Exercise 2 · A quadratic form

For $A = \begin{pmatrix}2 & 1\\1 & 3\end{pmatrix}$ and $f(x) = x^{\top}Ax$ , find $\nabla f$ at $x = (1, 1)$ .

Show solution

Step 1. $A$ is symmetric, so $\nabla(x^{\top}Ax) = 2Ax$ .

Step 2. $Ax = (2 + 1,\; 1 + 3) = (3, 4)$ at $x = (1, 1)$ .

Answer: $\nabla f = (6, 8)$ .

Exercise 3 · Its Hessian

Find the Hessian of $f$ in Exercise 2.

Show solution

$2A = \begin{pmatrix}4 & 2\\2 & 6\end{pmatrix}$ , constant because $f$ is quadratic.

Exercise 4 · Component check

Find the gradient of $f(x_1, x_2) = x_1^2 + x_1x_2 + 2x_2^2$ at $(1, 2)$ directly and via the matrix rule.

Show solution

Directly: $(2x_1 + x_2,\; x_1 + 4x_2) = (4, 9)$ . Matrix: $A = \begin{pmatrix}1 & 0.5\\0.5 & 2\end{pmatrix}$ , $2Ax = 2(1 + 1, 0.5 + 4) = (4, 9)$ . ✓

Exercise 5 · A Jacobian

Find the Jacobian of $g(x) = (x_1x_2,\; x_1 + x_2^2)^{\top}$ .

Show solution

$\frac{\partial g}{\partial x^{\top}} = \begin{pmatrix}x_2 & x_1\\1 & 2x_2\end{pmatrix}$ .

Part 2 · Proofs

Exercise 6 · The linear rule

Prove $\partial(a^{\top}x)/\partial x = a$ .

Show solution

$a^{\top}x = \sum_j a_jx_j$ , so $\partial/\partial x_\ell = a_\ell$ . Stacking gives $a$ .

Exercise 7 · The quadratic rule

Prove $\partial(x^{\top}Ax)/\partial x = (A + A^{\top})x$ .

Show solution

$x^{\top}Ax = \sum_i\sum_j a_{ij}x_ix_j$ . Differentiating with respect to $x_\ell$ picks up terms with $i = \ell$ and with $j = \ell$ : $\sum_j a_{\ell j}x_j + \sum_i a_{i\ell}x_i = (Ax)_\ell + (A^{\top}x)_\ell$ .

Exercise 8 · The normal equations

Derive $\nabla S(\beta)$ and the normal equations.

Show solution

Step 1. Expand $S(\beta) = y^{\top}y - 2\beta^{\top}X^{\top}y + \beta^{\top}X^{\top}X\beta$ (the two cross terms are equal scalars).

Step 2. By the rules, $\nabla S = -2X^{\top}y + 2X^{\top}X\beta$ .

Step 3. Setting it to zero: $X^{\top}X\hat\beta = X^{\top}y$ , so $\hat\beta = (X^{\top}X)^{-1}X^{\top}y$ under full rank.

Exercise 9 · A unique minimum

Show that the Hessian of $S$ is PD when $X$ has full column rank, so the solution is the unique minimiser.

Show solution

Step 1. $\nabla^2S = 2X^{\top}X$ and $v^{\top}X^{\top}Xv = \|Xv\|^2 > 0$ for $v \ne 0$ under full rank (Day 18).

Step 2. A function with a PD Hessian everywhere is strictly convex, so its stationary point is the unique global minimum (Day 25).

Exercise 10 · Ridge regression

Derive the minimiser of $S(\beta) + \lambda\|\beta\|^2$ and show it exists even when $X$ is rank-deficient (for $\lambda > 0$ ).

Show solution

Step 1. Gradient: $-2X^{\top}y + 2X^{\top}X\beta + 2\lambda\beta = 0 \Rightarrow (X^{\top}X + \lambda I)\beta = X^{\top}y$ .

Step 2. For $v \ne 0$ , $v^{\top}(X^{\top}X + \lambda I)v = \|Xv\|^2 + \lambda\|v\|^2 > 0$ , so the matrix is PD and invertible whatever the rank of $X$ .

Part 3 · Applied

Exercise 11 · A gradient from the farm data

For the four farms of Day 17 (visits $x$ , yields $y$ ), $X^{\top}X = \begin{pmatrix}4 & 10\\10 & 30\end{pmatrix}$ and $X^{\top}y = (16, 47)$ . Compute $\nabla S$ at $\beta = (0, 1)$ (no intercept, 1 t/ha per visit) and at $\hat\beta = (0.5, 1.4)$ .

Show solution

Step 1. $\nabla S(\beta) = -2(X^{\top}y - X^{\top}X\beta)$ .

Step 2. At $(0, 1)$ : $X^{\top}X\beta = (10, 30)$ , so $\nabla S = -2\big((16, 47) - (10, 30)\big) = -2(6, 17) = (-12, -34)$ .

Step 3. Both entries are negative, so increasing either coefficient lowers the SSR.

Step 4. At $\hat\beta$ : $X^{\top}X\hat\beta = (4(0.5) + 10(1.4),\; 10(0.5) + 30(1.4)) = (16, 47)$ , so $\nabla S = 0$ .

Answer: $(-12, -34)$ at the guess, and zero at the OLS estimate, as the normal equations require.

Exercise 12 · Weighted least squares

Derive the WLS estimator, and explain how weights $w_i = 1/\hat e(x_i)$ for treated and $1/(1 - \hat e(x_i))$ for control units turn a regression of $y$ on $(1, T)$ into an IPW estimator.

Show solution

Step 1. Objective: $(y - X\beta)^{\top}W(y - X\beta)$ with $W$ diagonal (one weight per unit).

Step 2. Gradient: $-2X^{\top}W(y - X\beta)$ . Setting it to zero gives $X^{\top}WX\beta = X^{\top}Wy$ , so $\hat\beta = (X^{\top}WX)^{-1}X^{\top}Wy$ .

Step 3. With $X = (\mathbf 1, T)$ , the fitted values are the weighted group means, so the slope is (weighted mean of treated outcomes) − (weighted mean of control outcomes).

Step 4. With weights $1/\hat e(x_i)$ for treated and $1/(1 - \hat e(x_i))$ for controls, that is the Hájek IPW estimator.

Answer: an IPW estimate of the ATE is a weighted regression of $y$ on a treatment dummy (Days 14 and 44).

Exercise 13 · The Hessian of a propensity-score logit

For a logit model of programme participation, $\ell(\beta) = \sum_i[y_ix_i^{\top}\beta - \ln(1 + e^{x_i^{\top}\beta})]$ ( $y_i = 1$ if household $i$ participates). Derive the gradient and Hessian and show that $\ell$ is concave.

Show solution

Step 1. Derivative of the second term: $\dfrac{\partial}{\partial\beta}\ln(1 + e^{x^{\top}\beta}) = \dfrac{e^{x^{\top}\beta}}{1 + e^{x^{\top}\beta}}x = p\,x$ .

Step 2. Gradient: $\nabla\ell = \sum_i(y_i - p_i)x_i = X^{\top}(y - p)$ .

Step 3. Differentiate $-p_ix_i$ again using $\partial p_i/\partial\beta^{\top} = p_i(1 - p_i)x_i^{\top}$ : $\nabla^2\ell = -\sum_ip_i(1 - p_i)x_ix_i^{\top} = -X^{\top}VX$ with $V = \operatorname{diag}(p_i(1 - p_i))$ .

Step 4. For any $v$ , $v^{\top}X^{\top}VXv = \sum_ip_i(1 - p_i)(x_i^{\top}v)^2 \ge 0$ , so the Hessian is negative semi-definite.

Answer: $\ell$ is concave, so any stationary point is a global maximum and Newton's method is reliable (Day 25).

Exercise 14 · The MLE of the error variance in a wage regression

For a log-wage regression with normal errors, $\ell = -\frac n2\ln(2\pi\sigma^2) - \frac{S(\beta)}{2\sigma^2}$ , where $S$ is the sum of squared residuals. Find the MLE of $\sigma^2$ and compare it with $s^2$ .

Show solution

Step 1. Differentiate with respect to $\sigma^2$ (treat it as one variable): $\dfrac{\partial\ell}{\partial\sigma^2} = -\dfrac{n}{2\sigma^2} + \dfrac{S}{2\sigma^4}$ .

Step 2. Set to zero and multiply by $2\sigma^4$ : $-n\sigma^2 + S = 0$ .

Step 3. At $\hat\beta$ : $\hat\sigma^2 = S(\hat\beta)/n = e^{\top}e/n$ .

Answer: $e^{\top}e/n$ , which is biased downward; $s^2 = e^{\top}e/(n - k)$ is unbiased (Day 27).

Exercise 15 · How much does one farm move the estimates?

$\hat\beta = (X^{\top}X)^{-1}X^{\top}y$ is linear in $y$ , so $\partial\hat\beta/\partial y^{\top} = (X^{\top}X)^{-1}X^{\top}$ . For the four farms, by how much do the intercept and the yield-per-visit slope change if farm 4's yield is recorded 1 t/ha higher?

Show solution

Step 1. $(X^{\top}X)^{-1} = \frac{1}{20}\begin{pmatrix}30 & -10\\-10 & 4\end{pmatrix}$ (determinant $4 \cdot 30 - 10^2 = 20$ ).

Step 2. Farm 4's column of $X^{\top}$ is $(1, 4)^{\top}$ .

Step 3. $\frac{1}{20}\begin{pmatrix}30 - 40\\-10 + 16\end{pmatrix} = \frac{1}{20}\begin{pmatrix}-10\\6\end{pmatrix} = \begin{pmatrix}-0.5\\0.3\end{pmatrix}$ .

Answer: the slope rises by 0.3 t/ha per visit and the intercept falls by 0.5: an end-point farm tilts the line. A data-entry error there matters a lot.

Verify in Python

import numpy as np

def num_grad(f, x, h=1e-6):
    return np.array([(f(x + h * e) - f(x - h * e)) / (2 * h) for e in np.eye(len(x))])

A = np.array([[2, 1], [1, 3.]])
print("grad x'Ax at (1,1):", num_grad(lambda x: x @ A @ x, np.array([1, 1.])).round(4))

x = np.array([1, 2, 3, 4.]); y = np.array([2, 3, 5, 6.]); X = np.column_stack([np.ones(4), x])
S = lambda b: (y - X @ b) @ (y - X @ b)
for b in (np.array([0, 1.]), np.array([0.5, 1.4])):
    print("grad S at", b, ":", num_grad(S, b).round(4), " formula:", (-2 * X.T @ (y - X @ b)).round(4))

print("dbeta/dy4:", (np.linalg.inv(X.T @ X) @ X.T)[:, 3])
lam = 0.5
print("ridge:", np.linalg.solve(X.T @ X + lam * np.eye(2), X.T @ y).round(4))

Key takeaways

$\partial(a^{\top}x)/\partial x = a$ and $\partial(x^{\top}Ax)/\partial x = 2Ax$ (symmetric $A$ ) do most of the work.
Setting the gradient of the SSR to zero gives the normal equations; its Hessian $2X^{\top}X$ guarantees a minimum.
Ridge, WLS and logit follow the same recipe: write the objective, differentiate, solve.

Day 24. Matrix Calculus & Gradients

Theory

Layout

Four rules cover almost everything

The least-squares objective

The same template, other estimators

Part 1 · Warm-up

Exercise 1 · A linear function

Exercise 2 · A quadratic form

Exercise 3 · Its Hessian

Exercise 4 · Component check

Exercise 5 · A Jacobian

Part 2 · Proofs

Exercise 6 · The linear rule

Exercise 7 · The quadratic rule

Exercise 8 · The normal equations

Exercise 9 · A unique minimum

Exercise 10 · Ridge regression

Part 3 · Applied

Exercise 11 · A gradient from the farm data

Exercise 12 · Weighted least squares

Exercise 13 · The Hessian of a propensity-score logit

Exercise 14 · The MLE of the error variance in a wage regression

Exercise 15 · How much does one farm move the estimates?

Verify in Python

Key takeaways

Simulation Lab: The Gradient of the Sum of Squares