{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Day 16: Vectors, Matrices & the Dot Product\n",
    "\n",
    "*Beginner to Ninja: Math for Causal Inference*\n",
    "\n",
    "Data as vectors, and the dot product as the engine behind means, covariances and correlations.\n",
    "\n",
    "**Learning outcomes**\n",
    "\n",
    "- Add, scale and take dot products of vectors, and compute norms and angles.\n",
    "- Prove the Cauchy–Schwarz inequality and the Pythagorean identity for orthogonal vectors.\n",
    "- Write means, covariances and correlations as dot products of (centred) vectors.\n",
    "- Lay out a data set as a design matrix with units in rows and variables in columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Theory\n",
    "\n",
    "### Why linear algebra now?\n",
    "\n",
    "Phase 1 worked with one or two variables at a time. Real causal analyses adjust for many covariates at once, and the cleanest language for \"many variables at once\" is linear algebra. By Day 27 every regression formula you know will be a one-line matrix expression.\n",
    "\n",
    "### Vectors\n",
    "\n",
    "A **vector** $x \\in \\mathbb{R}^n$ is an ordered list of $n$ numbers, written as a column. In data work, a vector usually holds one variable for all $n$ units (e.g. everyone's wage).\n",
    "\n",
    "- **Addition** and **scalar multiplication** work entry by entry: $(x + y)_i = x_i + y_i$, $(cx)_i = c\\,x_i$.\n",
    "- The **dot product** (inner product) is\n",
    "\n",
    "$$\n",
    "x \\cdot y = x^{\\top}y = \\sum_{i=1}^n x_iy_i.\n",
    "$$\n",
    "\n",
    "- The **norm** (length) is $\\|x\\| = \\sqrt{x^{\\top}x}$, and the **angle** $\\theta$ between $x$ and $y$ satisfies\n",
    "\n",
    "$$\n",
    "\\cos\\theta = \\frac{x^{\\top}y}{\\|x\\|\\,\\|y\\|}.\n",
    "$$\n",
    "\n",
    "- $x$ and $y$ are **orthogonal** when $x^{\\top}y = 0$.\n",
    "- **Cauchy–Schwarz:** $|x^{\\top}y| \\le \\|x\\|\\,\\|y\\|$, so $\\cos\\theta \\in [-1, 1]$.\n",
    "\n",
    "### Statistics as dot products\n",
    "\n",
    "Let $\\mathbf{1}$ be the vector of ones. Then\n",
    "\n",
    "$$\n",
    "\\bar x = \\tfrac1n\\mathbf{1}^{\\top}x, \\qquad \\tilde x = x - \\bar x\\mathbf{1} \\;(\\text{centred}), \\qquad\n",
    "\\widehat{\\operatorname{Cov}}(x, y) = \\tfrac{1}{n-1}\\tilde x^{\\top}\\tilde y, \\qquad r_{xy} = \\frac{\\tilde x^{\\top}\\tilde y}{\\|\\tilde x\\|\\,\\|\\tilde y\\|} = \\cos\\theta.\n",
    "$$\n",
    "\n",
    "**A correlation is the cosine of the angle between two centred data vectors.** That is why it lies in $[-1, 1]$.\n",
    "\n",
    "### Matrices\n",
    "\n",
    "An $n \\times k$ **matrix** is a rectangular array with $n$ rows and $k$ columns. The **transpose** $A^{\\top}$ swaps rows and columns. Special cases: the **identity** $I$ (ones on the diagonal), **diagonal** matrices, and **symmetric** matrices ($A = A^{\\top}$).\n",
    "\n",
    "The **design matrix** $X$ stacks the data: row $i$ is unit $i$, column $j$ is variable $j$, and the first column is usually $\\mathbf{1}$ (the intercept).\n",
    "\n",
    "> **Causal link.** Matching estimators (Phase 6) compare units whose covariate *vectors* are close; \"close\" means a small norm of the difference. Today's norm is the first distance you will use.\n",
    "\n",
    "## Part 1 · Warm-up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 · Vector arithmetic\n",
    "\n",
    "Let $u = (2, -1, 3)^{\\top}$ and $v = (1, 4, 0)^{\\top}$. Compute $u + v$ and $2u - v$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "$u + v = (3, 3, 3)^{\\top}$ and $2u - v = (4 - 1, -2 - 4, 6 - 0)^{\\top} = (3, -6, 6)^{\\top}$.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 · Dot product\n",
    "\n",
    "Compute $u^{\\top}v$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "$u^{\\top}v = 2(1) + (-1)(4) + 3(0) = -2$.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 · Norms\n",
    "\n",
    "Compute $\\|u\\|$ and $\\|v\\|$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "$\\|u\\| = \\sqrt{4 + 1 + 9} = \\sqrt{14} \\approx 3.742$ and $\\|v\\| = \\sqrt{1 + 16 + 0} = \\sqrt{17} \\approx 4.123$.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4 · The angle\n",
    "\n",
    "Find the angle between $u$ and $v$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** $\\cos\\theta = \\dfrac{u^{\\top}v}{\\|u\\|\\,\\|v\\|} = \\dfrac{-2}{\\sqrt{14}\\sqrt{17}} = \\dfrac{-2}{\\sqrt{238}} \\approx \\dfrac{-2}{15.43} \\approx -0.130$.\n",
    "\n",
    "**Step 2.** $\\theta = \\arccos(-0.130) \\approx 97.4°$.\n",
    "\n",
    "**Answer:** about 97°, slightly more than a right angle (the dot product is slightly negative).\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5 · Shapes\n",
    "\n",
    "$X$ is $4 \\times 3$. What are the shapes of $X^{\\top}$, $X^{\\top}X$ and $XX^{\\top}$?\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "$X^{\\top}$ is $3 \\times 4$; $X^{\\top}X$ is $3 \\times 3$ (one row and column per variable); $XX^{\\top}$ is $4 \\times 4$ (one per unit).\n",
    "\n",
    "</details>\n",
    "\n",
    "## Part 2 · Proofs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6 · Properties of the dot product\n",
    "\n",
    "Prove that $x^{\\top}y = y^{\\top}x$ and $x^{\\top}(y + z) = x^{\\top}y + x^{\\top}z$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "$x^{\\top}y = \\sum_i x_iy_i = \\sum_i y_ix_i = y^{\\top}x$, because real multiplication commutes. And $x^{\\top}(y + z) = \\sum_i x_i(y_i + z_i) = \\sum_i x_iy_i + \\sum_i x_iz_i$.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 7 · Pythagoras\n",
    "\n",
    "Prove $\\|x + y\\|^2 = \\|x\\|^2 + \\|y\\|^2 + 2x^{\\top}y$, and deduce the Pythagorean theorem for orthogonal vectors.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** Write the squared norm as a dot product and expand:\n",
    "\n",
    "$$\n",
    "\\|x + y\\|^2 = (x + y)^{\\top}(x + y) = x^{\\top}x + x^{\\top}y + y^{\\top}x + y^{\\top}y.\n",
    "$$\n",
    "\n",
    "**Step 2.** $x^{\\top}y = y^{\\top}x$ (Exercise 6), so the middle terms combine: $\\|x + y\\|^2 = \\|x\\|^2 + 2x^{\\top}y + \\|y\\|^2$.\n",
    "\n",
    "**Step 3.** If $x^{\\top}y = 0$, the cross term vanishes: $\\|x + y\\|^2 = \\|x\\|^2 + \\|y\\|^2$ (Pythagoras).\n",
    "\n",
    "This identity is behind the decomposition TSS = ESS + RSS on Day 20.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 8 · Cauchy–Schwarz\n",
    "\n",
    "Prove $|x^{\\top}y| \\le \\|x\\|\\,\\|y\\|$. *Hint: $\\|x - ty\\|^2 \\ge 0$ for every $t$.*\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** If $y = 0$ both sides are 0.\n",
    "\n",
    "**Step 2.** Otherwise $q(t) = \\|x - ty\\|^2 = \\|x\\|^2 - 2t\\,x^{\\top}y + t^2\\|y\\|^2 \\ge 0$ for all $t$.\n",
    "\n",
    "**Step 3.** A quadratic in $t$ that never goes negative has discriminant $\\le 0$:\n",
    "\n",
    "$$\n",
    "4(x^{\\top}y)^2 - 4\\|x\\|^2\\|y\\|^2 \\le 0.\n",
    "$$\n",
    "\n",
    "**Step 4.** (This is Day 5's proof that $|\\rho| \\le 1$, written for vectors.)\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 9 · Centred vectors are orthogonal to the constant\n",
    "\n",
    "Show that $\\bar x = \\frac1n\\mathbf{1}^{\\top}x$ and that $\\mathbf{1}^{\\top}\\tilde x = 0$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** $\\mathbf{1}^{\\top}x = \\sum_i x_i$, so dividing by $n$ gives the mean.\n",
    "\n",
    "**Step 2.** Then $\\mathbf{1}^{\\top}\\tilde x = \\mathbf{1}^{\\top}x - \\bar x\\,\\mathbf{1}^{\\top}\\mathbf{1} = n\\bar x - n\\bar x = 0$.\n",
    "\n",
    "**Step 3.** Deviations from the mean sum to zero; geometrically, the centred vector is perpendicular to $\\mathbf{1}$.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 10 · Correlation is a cosine\n",
    "\n",
    "Show that the sample correlation equals the cosine of the angle between $\\tilde x$ and $\\tilde y$, and conclude $|r| \\le 1$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** $r = \\frac{\\sum(x_i - \\bar x)(y_i - \\bar y)}{\\sqrt{\\sum(x_i - \\bar x)^2}\\sqrt{\\sum(y_i - \\bar y)^2}} = \\frac{\\tilde x^{\\top}\\tilde y}{\\|\\tilde x\\|\\,\\|\\tilde y\\|} = \\cos\\theta$\n",
    "\n",
    "**Step 2.** (the factors $1/(n-1)$ cancel).\n",
    "\n",
    "**Step 3.** Cauchy–Schwarz gives $|r| \\le 1$; $r = \\pm1$ exactly when $\\tilde y$ is a multiple of $\\tilde x$.\n",
    "\n",
    "</details>\n",
    "\n",
    "## Part 3 · Applied"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 11 · Revenue\n",
    "\n",
    "A firm sells three goods at prices $p = (2, 5, 10)$ in quantities $q = (100, 40, 10)$. Write revenue as a dot product and compute it.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** Revenue is $\\sum_j p_jq_j = p^{\\top}q$.\n",
    "\n",
    "**Step 2.** $2(100) + 5(40) + 10(10) = 200 + 200 + 100$.\n",
    "\n",
    "**Answer:** \\$500.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 12 · A weighted average\n",
    "\n",
    "A programme budget is split $w = (0.5, 0.3, 0.2)$ across three sites whose returns are $r = (4, -2, 10)$ percent. Find the weighted return.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** Weighted return $= w^{\\top}r = 0.5(4) + 0.3(-2) + 0.2(10)$.\n",
    "\n",
    "**Step 2.** $= 2 - 0.6 + 2 = 3.4$.\n",
    "\n",
    "**Answer:** 3.4%. Any weighted average (IPW, synthetic control) is a dot product with weights that sum to one.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 13 · Tutoring hours and test-score gains\n",
    "\n",
    "Four pupils received $x = (1, 2, 3, 6)$ hours of tutoring per week and gained $y = (2, 3, 5, 10)$ points on a test. Compute the centred vectors, the correlation and the regression slope $\\tilde x^{\\top}\\tilde y/\\tilde x^{\\top}\\tilde x$.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** Means: $\\bar x = 12/4 = 3$ hours and $\\bar y = 20/4 = 5$ points.\n",
    "\n",
    "**Step 2.** Centred vectors: $\\tilde x = (-2, -1, 0, 3)$ and $\\tilde y = (-3, -2, 0, 5)$.\n",
    "\n",
    "**Step 3.** Dot product and squared norms:\n",
    "\n",
    "$$\n",
    "\\tilde x^{\\top}\\tilde y = 6 + 2 + 0 + 15 = 23, \\qquad \\|\\tilde x\\|^2 = 4 + 1 + 0 + 9 = 14, \\qquad \\|\\tilde y\\|^2 = 9 + 4 + 0 + 25 = 38.\n",
    "$$\n",
    "\n",
    "**Step 4.** Correlation (cosine): $r = 23/\\sqrt{14 \\times 38} = 23/\\sqrt{532} \\approx 0.997$.\n",
    "\n",
    "**Step 5.** Slope: $23/14 \\approx 1.64$.\n",
    "\n",
    "**Answer:** $r \\approx 0.997$; each extra weekly hour goes with about 1.6 more points (an association, not yet a causal effect).\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 14 · A design matrix\n",
    "\n",
    "Three workers have education 12, 16 and 14 years; workers 2 and 3 were trained. Write the design matrix for a regression of wages on an intercept, education and training.\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** One row per worker, in the order given.\n",
    "\n",
    "**Step 2.** Columns: a column of ones (intercept), years of education, and the training dummy (0 for worker 1, 1 for workers 2 and 3).\n",
    "\n",
    "**Answer:**\n",
    "\n",
    "$$\n",
    "X = \\begin{pmatrix} 1 & 12 & 0 \\\\ 1 & 16 & 1 \\\\ 1 & 14 & 1 \\end{pmatrix}.\n",
    "$$\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 15 · Distance between units\n",
    "\n",
    "Firm A has covariates (size 10, age 2) and firm B (size 13, age 6). Find their Euclidean distance. What happens if size is measured in hundreds instead of units?\n",
    "\n",
    "<details><summary>Show solution</summary>\n",
    "\n",
    "**Step 1.** Difference in covariates: $b - a = (3, 4)$.\n",
    "\n",
    "**Step 2.** Distance: $\\sqrt{3^2 + 4^2} = 5$.\n",
    "\n",
    "**Step 3.** In hundreds, sizes become 0.10 and 0.13, so the size difference is 0.03 and the distance is $\\sqrt{0.03^2 + 4^2} \\approx 4.0$.\n",
    "\n",
    "**Answer:** 5, but about 4.0 after rescaling: the distance is now driven almost entirely by age. Euclidean distance depends on units, which is why matching uses standardised or Mahalanobis distances (Day 21).\n",
    "\n",
    "</details>\n",
    "\n",
    "## Verify in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your work:*"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "u, v = np.array([2, -1, 3]), np.array([1, 4, 0])\n",
    "print(\"u+v =\", u + v, \" 2u-v =\", 2 * u - v, \" u.v =\", u @ v)\n",
    "cos = u @ v / (np.linalg.norm(u) * np.linalg.norm(v))\n",
    "print(f\"norms {np.linalg.norm(u):.3f}, {np.linalg.norm(v):.3f}; angle {np.degrees(np.arccos(cos)):.1f} degrees\")\n",
    "\n",
    "x, y = np.array([1, 2, 3, 6.]), np.array([2, 3, 5, 10.])\n",
    "xc, yc = x - x.mean(), y - y.mean()\n",
    "print(\"r (cosine) =\", round(xc @ yc / np.sqrt((xc @ xc) * (yc @ yc)), 4), \" numpy:\", round(np.corrcoef(x, y)[0, 1], 4))\n",
    "print(\"slope =\", round(xc @ yc / (xc @ xc), 3), \" 1'xc =\", xc.sum())\n",
    "\n",
    "X = np.array([[1, 12, 0], [1, 16, 1], [1, 14, 1]])\n",
    "print(\"X'X =\\n\", X.T @ X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Key takeaways\n",
    "\n",
    "- A data set is a matrix; each variable is a vector; most statistics are dot products.\n",
    "- Correlation is the cosine of the angle between centred vectors, so Cauchy–Schwarz bounds it by 1.\n",
    "- Orthogonal vectors obey Pythagoras, the identity behind every variance decomposition in regression."
   ]
  }
 ]
}