{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 15: Phase 1 Capstone: Simpson's Paradox, Conditional Independence & Selection Bias\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Pulling Phase 1 together: when do comparisons in data answer causal questions?\n", "\n", "**Learning outcomes**\n", "\n", "- Define conditional independence and show it neither implies nor is implied by marginal independence.\n", "- Explain Simpson's paradox algebraically and resolve it with standardisation.\n", "- Derive the decomposition of a naive comparison into the ATT plus selection bias.\n", "- Recognise collider (Berkson) bias and choose the right Phase 1 tool for a problem." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Conditional independence\n", "\n", "$X$ and $Y$ are **conditionally independent given $Z$**, written $X \\perp Y \\mid Z$, if\n", "\n", "$$\n", "P(X, Y \\mid Z) = P(X \\mid Z)\\,P(Y \\mid Z) \\quad\\Longleftrightarrow\\quad P(X \\mid Y, Z) = P(X \\mid Z).\n", "$$\n", "\n", "Two surprises that drive much of causal inference:\n", "\n", "- Variables can be **dependent overall but independent within strata** (a common cause $Z$ explains their association).\n", "- Variables can be **independent overall but dependent within strata** (conditioning on a common *effect*, a \"collider\", creates an association: Berkson's paradox).\n", "\n", "Phase 4 turns these patterns into graphs (DAGs and d-separation). Today we stay with algebra.\n", "\n", "### Simpson's paradox\n", "\n", "By the law of total probability,\n", "\n", "$$\n", "P(Y = 1 \\mid T = t) = \\sum_z P(Y = 1 \\mid T = t, Z = z)\\,P(Z = z \\mid T = t).\n", "$$\n", "\n", "The overall rate for each arm averages its stratum rates with *its own* weights $P(Z \\mid T = t)$. If the arms have different case mixes, the overall comparison can reverse the within-stratum comparison. **Standardising** uses common weights $P(Z = z)$ instead, which is Day 6's adjustment formula.\n", "\n", "### Selection bias decomposition\n", "\n", "With potential outcomes $Y(1), Y(0)$ and observed $Y = Y(T)$:\n", "\n", "$$\n", "\\underbrace{E[Y \\mid T = 1] - E[Y \\mid T = 0]}_{\\text{naive comparison}} = \\underbrace{E[Y(1) - Y(0) \\mid T = 1]}_{\\text{ATT}} + \\underbrace{E[Y(0) \\mid T = 1] - E[Y(0) \\mid T = 0]}_{\\text{selection bias}}.\n", "$$\n", "\n", "Selection bias is the gap in *untreated* outcomes between those who did and did not get treatment. It is zero when $T \\perp Y(0)$, which randomisation guarantees. When we can only claim $\\big(Y(1), Y(0)\\big) \\perp T \\mid Z$ (**unconfoundedness**), the decomposition holds within each stratum of $Z$, and we standardise.\n", "\n", "### Your Phase 1 toolkit\n", "\n", "| Question | Tool (Day) |\n", "|---|---|\n", "| What can I say about a joint probability from marginals? | Fréchet bounds (1) |\n", "| How do overall and subgroup rates relate? | Law of total probability (2), LIE (6) |\n", "| Is the effect an average of something? | Linearity of expectation (3) |\n", "| How does omitting a variable bias a slope? | Covariance algebra (5) |\n", "| How precise is my estimate? | CLT, SE, CI, bootstrap (7–9) |\n", "| Could this be noise? Can my study detect it? | Tests and power (10) |\n", "| How do I fit a model / a propensity score? | Maximum likelihood (11) |\n", "| SE of a ratio or % effect? | Delta method (12) |\n", "| How do I combine prior evidence or shrink noisy estimates? | Bayes (13) |\n", "| Some outcomes are unobserved | MAR, IPW (14) |\n", "\n", "## Part 1 · Warm-up\n", "\n", "The classic kidney-stone study (Charig et al., 1986) compared open surgery (A) with percutaneous nephrolithotomy (B):\n", "\n", "| | Small stones | Large stones | All |\n", "|---|---|---|---|\n", "| A: successes / patients | 81 / 87 | 192 / 263 | 273 / 350 |\n", "| B: successes / patients | 234 / 270 | 55 / 80 | 289 / 350 |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Stratum rates\n", "\n", "Compute each treatment's success rate for small and large stones.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Small stones: A $81/87 \\approx 93.1\\%$; B $234/270 \\approx 86.7\\%$.\n", "\n", "**Step 2.** Large stones: A $192/263 \\approx 73.0\\%$; B $55/80 \\approx 68.8\\%$.\n", "\n", "**Answer:** A has the higher success rate for both stone sizes.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Overall rates\n", "\n", "Compute the overall success rates.\n", "\n", "
Show solution\n", "\n", "**Step 1.** A: $273/350 = 78.0\\%$.\n", "\n", "**Step 2.** B: $289/350 \\approx 82.6\\%$.\n", "\n", "**Answer:** B looks better overall although A is better for each stone size: Simpson's paradox.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · The weights\n", "\n", "What share of each treatment's patients had large stones? Explain the reversal.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Share with large stones: A $263/350 \\approx 75\\%$; B $80/350 \\approx 23\\%$.\n", "\n", "**Step 2.** Large stones are harder to treat (success ~70% versus ~90%), and A got mostly large stones.\n", "\n", "**Answer:** each overall rate averages that treatment's stratum rates with its own case mix. A's average leans on the difficult stratum, which drags it below B's.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Dependent overall, independent within strata\n", "\n", "$Z \\sim \\text{Bern}(0.5)$. Given $Z$, $X$ and $Y$ are independent Bernoulli($0.2 + 0.6Z$). Are $X$ and $Y$ independent?\n", "\n", "
Show solution\n", "\n", "**Step 1.** By construction $X \\perp Y \\mid Z$.\n", "\n", "**Step 2.** Marginally, $P(X = 1) = P(Y = 1) = 0.5$, but $P(X = 1, Y = 1) = 0.5(0.2^2) + 0.5(0.8^2) = 0.34 \\ne 0.25$.\n", "\n", "**Step 3.** They are dependent: the common cause $Z$ links them.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Independent overall, dependent within strata\n", "\n", "Talent $T$ and charm $C$ are independent Bernoulli(0.5). A firm hires anyone with at least one. Among hires, compute $P(T = 1)$ and $P(T = 1 \\mid C = 1)$.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Hired = $\\{T = 1 \\text{ or } C = 1\\}$, probability 0.75.\n", "\n", "**Step 2.** $P(T = 1 \\mid \\text{hired}) = 0.5/0.75 = 2/3$.\n", "\n", "**Step 3.** Among hires with charm, talent is 1 with probability $0.25/0.5 = 1/2$.\n", "\n", "**Step 4.** So among hires, charm and talent are *negatively* related, although they are independent in the population.\n", "\n", "**Step 5.** Conditioning on a common effect (hiring) created the association.\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · The selection-bias decomposition\n", "\n", "Prove the decomposition of the naive comparison into ATT plus selection bias.\n", "\n", "
Show solution\n", "\n", "$E[Y \\mid T = 1] = E[Y(1) \\mid T = 1]$ and $E[Y \\mid T = 0] = E[Y(0) \\mid T = 0]$. Add and subtract $E[Y(0) \\mid T = 1]$:\n", "\n", "$$\n", "E[Y(1) \\mid T = 1] - E[Y(0) \\mid T = 1] + E[Y(0) \\mid T = 1] - E[Y(0) \\mid T = 0].\n", "$$\n", "\n", "The first pair is the ATT (by linearity); the second pair is selection bias.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Randomisation removes selection bias\n", "\n", "Show that if $T \\perp \\big(Y(1), Y(0)\\big)$, the naive comparison equals the ATE.\n", "\n", "
Show solution\n", "\n", "Independence gives $E[Y(0) \\mid T = 1] = E[Y(0) \\mid T = 0] = E[Y(0)]$, so selection bias is zero. It also gives $E[Y(1) - Y(0) \\mid T = 1] = E[Y(1) - Y(0)]$, so ATT = ATE.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · The adjustment formula\n", "\n", "Prove that if $\\big(Y(1), Y(0)\\big) \\perp T \\mid Z$ and $0 < P(T = 1 \\mid Z) < 1$, then $E[Y(1)] = \\sum_z E[Y \\mid T = 1, Z = z]P(Z = z)$.\n", "\n", "
Show solution\n", "\n", "$$\n", "E[Y(1)] = \\sum_z E[Y(1) \\mid Z = z]P(z) = \\sum_z E[Y(1) \\mid T = 1, Z = z]P(z) = \\sum_z E[Y \\mid T = 1, Z = z]P(z).\n", "$$\n", "\n", "The first step is the LIE, the second unconfoundedness, the third that $Y = Y(1)$ when $T = 1$. Overlap ensures $E[Y \\mid T = 1, Z = z]$ is defined for every $z$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · No reversal without imbalance\n", "\n", "Show that if $P(Z \\mid T = 1) = P(Z \\mid T = 0)$ and A is better in every stratum, A is better overall.\n", "\n", "
Show solution\n", "\n", "**Step 1.** With common weights $w_z$, $P(Y \\mid A) - P(Y \\mid B) = \\sum_z w_z\\big[P(Y \\mid A, z) - P(Y \\mid B, z)\\big]$, a weighted sum of positive numbers with non-negative weights summing to 1, hence positive.\n", "\n", "**Step 2.** Simpson's reversal needs the case mix to differ between arms, which randomisation prevents in expectation.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Two forms of conditional independence\n", "\n", "Show that $P(X, Y \\mid Z) = P(X \\mid Z)P(Y \\mid Z)$ implies $P(X \\mid Y, Z) = P(X \\mid Z)$ (where defined).\n", "\n", "
Show solution\n", "\n", "$$\n", "P(X \\mid Y, Z) = \\frac{P(X, Y \\mid Z)}{P(Y \\mid Z)} = \\frac{P(X \\mid Z)P(Y \\mid Z)}{P(Y \\mid Z)} = P(X \\mid Z).\n", "$$\n", "\n", "Once you know $Z$, learning $Y$ tells you nothing more about $X$.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Standardising the kidney-stone data\n", "\n", "Assume unconfoundedness given stone size and standardise to the pooled population (357 small and 343 large of 700). Which treatment is better, and by how much?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Common weights from the pooled population: small $357/700 = 0.51$, large $343/700 = 0.49$.\n", "\n", "**Step 2.** Standardised success for A:\n", "\n", "$$\n", "0.931(0.51) + 0.730(0.49) \\approx 0.475 + 0.358 = 0.833.\n", "$$\n", "\n", "**Step 3.** Standardised success for B:\n", "\n", "$$\n", "0.867(0.51) + 0.688(0.49) \\approx 0.442 + 0.337 = 0.779.\n", "$$\n", "\n", "**Step 4.** Difference: $0.833 - 0.779 \\approx 0.054$.\n", "\n", "**Answer:** with both treatments applied to the same case mix, A succeeds about 5.4 points more often, reversing the naive comparison.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Selection bias in job training\n", "\n", "Trainees earn \\$30k on average and non-trainees \\$34k. A credible study finds the ATT is +\\$2k. What is the selection bias, and what does it mean?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Naive comparison: $30 - 34 = -4$ (thousand dollars).\n", "\n", "**Step 2.** Decomposition: naive $=$ ATT $+$ selection bias, so $-4 = 2 + \\text{bias}$.\n", "\n", "**Step 3.** Selection bias $= -6$.\n", "\n", "**Answer:** −\\$6k. Without training, trainees would have earned \\$6k less than non-trainees, because the programme targets disadvantaged workers.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Hospital-based studies\n", "\n", "Two diseases are independent in the population, but each raises the chance of hospitalisation. A study of hospital patients finds them negatively associated. Explain.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Draw the structure: disease 1 → hospitalised ← disease 2. Hospitalisation is a common effect (a **collider**).\n", "\n", "**Step 2.** Among hospitalised patients, someone without disease 1 must have been admitted for another reason, so is more likely to have disease 2.\n", "\n", "**Step 3.** So within the hospital sample the diseases are negatively associated, though independent in the population (Berkson, 1946).\n", "\n", "**Answer:** selecting the sample on a common effect creates a spurious association. Do not condition on variables affected by both the exposure and the outcome.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Controlling for a post-treatment variable\n", "\n", "A training programme raises both job quality and wages. An analyst \"controls for\" job quality when estimating the effect on wages. What goes wrong?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Training → job quality → wages: job quality is a **mediator**.\n", "\n", "**Step 2.** Holding job quality fixed removes the part of the effect that works through better jobs, so the estimate is no longer the total effect.\n", "\n", "**Step 3.** If unobserved factors (e.g. motivation) affect both job quality and wages, conditioning on job quality also opens a collider path and adds bias.\n", "\n", "**Answer:** control only for pre-treatment variables when estimating a total effect.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Choose the tool\n", "\n", "For each task, name the Phase 1 tool: (a) a 95% interval for a median income; (b) the SE of a percentage effect from a log-point coefficient; (c) correcting a survey mean when response depends on age; (d) the sample size for 80% power; (e) estimating a propensity score.\n", "\n", "
Show solution\n", "\n", "**Step 1.** (a) Median income has no simple SE formula → **bootstrap** (Day 9).\n", "\n", "**Step 2.** (b) A function of an estimate → **delta method** (Day 12).\n", "\n", "**Step 3.** (c) Response depends on an observed covariate → **IPW under MAR** (Day 14).\n", "\n", "**Step 4.** (d) Planning a study → **power formula** (Day 10).\n", "\n", "**Step 5.** (e) A binary treatment model → **logit by maximum likelihood** (Day 11).\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "succ = {\"A\": (81, 87, 192, 263), \"B\": (234, 270, 55, 80)}\n", "w_small = (87 + 270) / 700\n", "for arm, (s1, n1, s2, n2) in succ.items():\n", " r1, r2 = s1 / n1, s2 / n2\n", " print(f\"{arm}: small {r1:.3f}, large {r2:.3f}, overall {(s1 + s2) / (n1 + n2):.3f}, \"\n", " f\"standardised {w_small * r1 + (1 - w_small) * r2:.3f}\")\n", "\n", "# Berkson's paradox by simulation\n", "rng = np.random.default_rng(15)\n", "talent = rng.random(1_000_000) < 0.5\n", "charm = rng.random(1_000_000) < 0.5\n", "hired = talent | charm\n", "print(\"corr in population:\", np.corrcoef(talent, charm)[0, 1].round(3),\n", " \" corr among hires:\", np.corrcoef(talent[hired], charm[hired])[0, 1].round(3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- A naive comparison = ATT + selection bias; randomisation sets the bias to zero.\n", "- Simpson's paradox comes from different case mixes; standardise with common weights.\n", "- Conditioning on a common cause can remove an association; conditioning on a common effect can create one." ] } ] }