Day 16: Vectors, Matrices & the Dot Product · Math for Causal Inference

Theory

Why linear algebra now?

Phase 1 worked with one or two variables at a time. Real causal analyses adjust for many covariates at once, and the cleanest language for "many variables at once" is linear algebra. By Day 27 every regression formula you know will be a one-line matrix expression.

Vectors

A vector $x \in \mathbb{R}^n$ is an ordered list of $n$ numbers, written as a column. In data work, a vector usually holds one variable for all $n$ units (e.g. everyone's wage).

Addition and scalar multiplication work entry by entry: $(x + y)_i = x_i + y_i$ , $(cx)_i = c\,x_i$ .
The dot product (inner product) is

x \cdot y = x^{\top}y = \sum_{i=1}^n x_iy_i.

The norm (length) is $\|x\| = \sqrt{x^{\top}x}$ , and the angle $\theta$ between $x$ and $y$ satisfies

\cos\theta = \frac{x^{\top}y}{\|x\|\,\|y\|}.

$x$ and $y$ are orthogonal when $x^{\top}y = 0$ .
Cauchy–Schwarz: $|x^{\top}y| \le \|x\|\,\|y\|$ , so $\cos\theta \in [-1, 1]$ .

Statistics as dot products

Let $\mathbf{1}$ be the vector of ones. Then

\bar x = \tfrac1n\mathbf{1}^{\top}x, \qquad \tilde x = x - \bar x\mathbf{1} \;(\text{centred}), \qquad \widehat{\operatorname{Cov}}(x, y) = \tfrac{1}{n-1}\tilde x^{\top}\tilde y, \qquad r_{xy} = \frac{\tilde x^{\top}\tilde y}{\|\tilde x\|\,\|\tilde y\|} = \cos\theta.

A correlation is the cosine of the angle between two centred data vectors. That is why it lies in $[-1, 1]$ .

Matrices

An $n \times k$ matrix is a rectangular array with $n$ rows and $k$ columns. The transpose $A^{\top}$ swaps rows and columns. Special cases: the identity $I$ (ones on the diagonal), diagonal matrices, and symmetric matrices ( $A = A^{\top}$ ).

The design matrix $X$ stacks the data: row $i$ is unit $i$ , column $j$ is variable $j$ , and the first column is usually $\mathbf{1}$ (the intercept).

Causal link. Matching estimators (Phase 6) compare units whose covariate vectors are close; "close" means a small norm of the difference. Today's norm is the first distance you will use.

Part 1 · Warm-up

Exercise 1 · Vector arithmetic

Let $u = (2, -1, 3)^{\top}$ and $v = (1, 4, 0)^{\top}$ . Compute $u + v$ and $2u - v$ .

Show solution

$u + v = (3, 3, 3)^{\top}$ and $2u - v = (4 - 1, -2 - 4, 6 - 0)^{\top} = (3, -6, 6)^{\top}$ .

Exercise 2 · Dot product

Compute $u^{\top}v$ .

Show solution

$u^{\top}v = 2(1) + (-1)(4) + 3(0) = -2$ .

Exercise 3 · Norms

Compute $\|u\|$ and $\|v\|$ .

Show solution

$\|u\| = \sqrt{4 + 1 + 9} = \sqrt{14} \approx 3.742$ and $\|v\| = \sqrt{1 + 16 + 0} = \sqrt{17} \approx 4.123$ .

Exercise 4 · The angle

Find the angle between $u$ and $v$ .

Show solution

Step 1. $\cos\theta = \dfrac{u^{\top}v}{\|u\|\,\|v\|} = \dfrac{-2}{\sqrt{14}\sqrt{17}} = \dfrac{-2}{\sqrt{238}} \approx \dfrac{-2}{15.43} \approx -0.130$ .

Step 2. $\theta = \arccos(-0.130) \approx 97.4°$ .

Answer: about 97°, slightly more than a right angle (the dot product is slightly negative).

Exercise 5 · Shapes

$X$ is $4 \times 3$ . What are the shapes of $X^{\top}$ , $X^{\top}X$ and $XX^{\top}$ ?

Show solution

$X^{\top}$ is $3 \times 4$ ; $X^{\top}X$ is $3 \times 3$ (one row and column per variable); $XX^{\top}$ is $4 \times 4$ (one per unit).

Part 2 · Proofs

Exercise 6 · Properties of the dot product

Prove that $x^{\top}y = y^{\top}x$ and $x^{\top}(y + z) = x^{\top}y + x^{\top}z$ .

Show solution

$x^{\top}y = \sum_i x_iy_i = \sum_i y_ix_i = y^{\top}x$ , because real multiplication commutes. And $x^{\top}(y + z) = \sum_i x_i(y_i + z_i) = \sum_i x_iy_i + \sum_i x_iz_i$ .

Exercise 7 · Pythagoras

Prove $\|x + y\|^2 = \|x\|^2 + \|y\|^2 + 2x^{\top}y$ , and deduce the Pythagorean theorem for orthogonal vectors.

Show solution

Step 1. Write the squared norm as a dot product and expand:

\|x + y\|^2 = (x + y)^{\top}(x + y) = x^{\top}x + x^{\top}y + y^{\top}x + y^{\top}y.

Step 2. $x^{\top}y = y^{\top}x$ (Exercise 6), so the middle terms combine: $\|x + y\|^2 = \|x\|^2 + 2x^{\top}y + \|y\|^2$ .

Step 3. If $x^{\top}y = 0$ , the cross term vanishes: $\|x + y\|^2 = \|x\|^2 + \|y\|^2$ (Pythagoras).

This identity is behind the decomposition TSS = ESS + RSS on Day 20.

Exercise 8 · Cauchy–Schwarz

Prove $|x^{\top}y| \le \|x\|\,\|y\|$ . Hint: $\|x - ty\|^2 \ge 0$ for every $t$ .

Show solution

Step 1. If $y = 0$ both sides are 0.

Step 2. Otherwise $q(t) = \|x - ty\|^2 = \|x\|^2 - 2t\,x^{\top}y + t^2\|y\|^2 \ge 0$ for all $t$ .

Step 3. A quadratic in $t$ that never goes negative has discriminant $\le 0$ :

4(x^{\top}y)^2 - 4\|x\|^2\|y\|^2 \le 0.

Step 4. (This is Day 5's proof that $|\rho| \le 1$ , written for vectors.)

Exercise 9 · Centred vectors are orthogonal to the constant

Show that $\bar x = \frac1n\mathbf{1}^{\top}x$ and that $\mathbf{1}^{\top}\tilde x = 0$ .

Show solution

Step 1. $\mathbf{1}^{\top}x = \sum_i x_i$ , so dividing by $n$ gives the mean.

Step 2. Then $\mathbf{1}^{\top}\tilde x = \mathbf{1}^{\top}x - \bar x\,\mathbf{1}^{\top}\mathbf{1} = n\bar x - n\bar x = 0$ .

Step 3. Deviations from the mean sum to zero; geometrically, the centred vector is perpendicular to $\mathbf{1}$ .

Exercise 10 · Correlation is a cosine

Show that the sample correlation equals the cosine of the angle between $\tilde x$ and $\tilde y$ , and conclude $|r| \le 1$ .

Show solution

Step 1. $r = \frac{\sum(x_i - \bar x)(y_i - \bar y)}{\sqrt{\sum(x_i - \bar x)^2}\sqrt{\sum(y_i - \bar y)^2}} = \frac{\tilde x^{\top}\tilde y}{\|\tilde x\|\,\|\tilde y\|} = \cos\theta$

Step 2. (the factors $1/(n-1)$ cancel).

Step 3. Cauchy–Schwarz gives $|r| \le 1$ ; $r = \pm1$ exactly when $\tilde y$ is a multiple of $\tilde x$ .

Part 3 · Applied

Exercise 11 · Revenue

A firm sells three goods at prices $p = (2, 5, 10)$ in quantities $q = (100, 40, 10)$ . Write revenue as a dot product and compute it.

Show solution

Step 1. Revenue is $\sum_j p_jq_j = p^{\top}q$ .

Step 2. $2(100) + 5(40) + 10(10) = 200 + 200 + 100$ .

Answer: $500.

Exercise 12 · A weighted average

A programme budget is split $w = (0.5, 0.3, 0.2)$ across three sites whose returns are $r = (4, -2, 10)$ percent. Find the weighted return.

Show solution

Step 1. Weighted return $= w^{\top}r = 0.5(4) + 0.3(-2) + 0.2(10)$ .

Step 2. $= 2 - 0.6 + 2 = 3.4$ .

Answer: 3.4%. Any weighted average (IPW, synthetic control) is a dot product with weights that sum to one.

Exercise 13 · Tutoring hours and test-score gains

Four pupils received $x = (1, 2, 3, 6)$ hours of tutoring per week and gained $y = (2, 3, 5, 10)$ points on a test. Compute the centred vectors, the correlation and the regression slope $\tilde x^{\top}\tilde y/\tilde x^{\top}\tilde x$ .

Show solution

Step 1. Means: $\bar x = 12/4 = 3$ hours and $\bar y = 20/4 = 5$ points.

Step 2. Centred vectors: $\tilde x = (-2, -1, 0, 3)$ and $\tilde y = (-3, -2, 0, 5)$ .

Step 3. Dot product and squared norms:

\tilde x^{\top}\tilde y = 6 + 2 + 0 + 15 = 23, \qquad \|\tilde x\|^2 = 4 + 1 + 0 + 9 = 14, \qquad \|\tilde y\|^2 = 9 + 4 + 0 + 25 = 38.

Step 4. Correlation (cosine): $r = 23/\sqrt{14 \times 38} = 23/\sqrt{532} \approx 0.997$ .

Step 5. Slope: $23/14 \approx 1.64$ .

Answer: $r \approx 0.997$ ; each extra weekly hour goes with about 1.6 more points (an association, not yet a causal effect).

Exercise 14 · A design matrix

Three workers have education 12, 16 and 14 years; workers 2 and 3 were trained. Write the design matrix for a regression of wages on an intercept, education and training.

Show solution

Step 1. One row per worker, in the order given.

Step 2. Columns: a column of ones (intercept), years of education, and the training dummy (0 for worker 1, 1 for workers 2 and 3).

Answer:

X = \begin{pmatrix} 1 & 12 & 0 \\ 1 & 16 & 1 \\ 1 & 14 & 1 \end{pmatrix}.

Exercise 15 · Distance between units

Firm A has covariates (size 10, age 2) and firm B (size 13, age 6). Find their Euclidean distance. What happens if size is measured in hundreds instead of units?

Show solution

Step 1. Difference in covariates: $b - a = (3, 4)$ .

Step 2. Distance: $\sqrt{3^2 + 4^2} = 5$ .

Step 3. In hundreds, sizes become 0.10 and 0.13, so the size difference is 0.03 and the distance is $\sqrt{0.03^2 + 4^2} \approx 4.0$ .

Answer: 5, but about 4.0 after rescaling: the distance is now driven almost entirely by age. Euclidean distance depends on units, which is why matching uses standardised or Mahalanobis distances (Day 21).

Verify in Python

import numpy as np

u, v = np.array([2, -1, 3]), np.array([1, 4, 0])
print("u+v =", u + v, " 2u-v =", 2 * u - v, " u.v =", u @ v)
cos = u @ v / (np.linalg.norm(u) * np.linalg.norm(v))
print(f"norms {np.linalg.norm(u):.3f}, {np.linalg.norm(v):.3f}; angle {np.degrees(np.arccos(cos)):.1f} degrees")

x, y = np.array([1, 2, 3, 6.]), np.array([2, 3, 5, 10.])
xc, yc = x - x.mean(), y - y.mean()
print("r (cosine) =", round(xc @ yc / np.sqrt((xc @ xc) * (yc @ yc)), 4), " numpy:", round(np.corrcoef(x, y)[0, 1], 4))
print("slope =", round(xc @ yc / (xc @ xc), 3), " 1'xc =", xc.sum())

X = np.array([[1, 12, 0], [1, 16, 1], [1, 14, 1]])
print("X'X =\n", X.T @ X)

Key takeaways

A data set is a matrix; each variable is a vector; most statistics are dot products.
Correlation is the cosine of the angle between centred vectors, so Cauchy–Schwarz bounds it by 1.
Orthogonal vectors obey Pythagoras, the identity behind every variance decomposition in regression.

Day 16. Vectors, Matrices & the Dot Product

Theory

Why linear algebra now?

Vectors

Statistics as dot products

Matrices

Part 1 · Warm-up

Exercise 1 · Vector arithmetic

Exercise 2 · Dot product

Exercise 3 · Norms

Exercise 4 · The angle

Exercise 5 · Shapes

Part 2 · Proofs

Exercise 6 · Properties of the dot product

Exercise 7 · Pythagoras

Exercise 8 · Cauchy–Schwarz

Exercise 9 · Centred vectors are orthogonal to the constant

Exercise 10 · Correlation is a cosine

Part 3 · Applied

Exercise 11 · Revenue

Exercise 12 · A weighted average

Exercise 13 · Tutoring hours and test-score gains

Exercise 14 · A design matrix

Exercise 15 · Distance between units

Verify in Python

Key takeaways

Simulation Lab: Vector Playground