{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Day 2: Conditional Probability, Independence & Bayes\n", "\n", "*Beginner to Ninja: Math for Causal Inference*\n", "\n", "Updating probabilities when you learn something, and the four rules that do it.\n", "\n", "**Learning outcomes**\n", "\n", "- Compute conditional probabilities and test events for independence.\n", "- Use the law of total probability to combine subgroup rates.\n", "- Derive and apply Bayes' theorem, including in low-prevalence settings.\n", "- Explain why aggregate comparisons can reverse within subgroups (Simpson's paradox preview)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Theory\n", "\n", "### Conditioning restricts the sample space\n", "\n", "Causal questions are almost never about raw probabilities. We want the probability of employment *given* training, or of recovery *given* a drug. Learning that $B$ happened shrinks the world to $B$:\n", "\n", "$$\n", "P(A \\mid B) = \\frac{P(A \\cap B)}{P(B)}, \\qquad P(B) > 0.\n", "$$\n", "\n", "Rearranged, this is the **multiplication rule**: $P(A \\cap B) = P(A \\mid B)\\,P(B)$.\n", "\n", "### Independence\n", "\n", "$A$ and $B$ are **independent** if\n", "\n", "$$\n", "P(A \\cap B) = P(A)\\,P(B) \\quad\\Longleftrightarrow\\quad P(A \\mid B) = P(A) \\;\\;(\\text{when } P(B) > 0).\n", "$$\n", "\n", "Knowing $B$ tells you nothing about $A$. Randomisation makes treatment independent of *every* pre-treatment characteristic, observed or not. That single fact is why experiments identify causal effects.\n", "\n", "### Law of total probability (LTP)\n", "\n", "If $B_1, \\dots, B_k$ partition $\\Omega$:\n", "\n", "$$\n", "P(A) = \\sum_{j=1}^{k} P(A \\mid B_j)\\,P(B_j).\n", "$$\n", "\n", "An overall rate is a **weighted average** of subgroup rates, with weights equal to subgroup shares. Keep this in mind: when two groups have different weights, their overall rates can mislead.\n", "\n", "### Bayes' theorem\n", "\n", "$$\n", "P(A \\mid B) = \\frac{P(B \\mid A)\\,P(A)}{P(B)} = \\frac{P(B \\mid A)\\,P(A)}{P(B \\mid A)P(A) + P(B \\mid A^c)P(A^c)}.\n", "$$\n", "\n", "Bayes reverses the direction of conditioning: from \"how likely is the evidence given the hypothesis\" to \"how likely is the hypothesis given the evidence\". The **base rate** $P(A)$ matters enormously.\n", "\n", "> **Causal link.** \"Treatment is independent of potential outcomes\" is the key identifying assumption of an experiment. Today's definition of independence is what that sentence means.\n", "\n", "## Part 1 · Warm-up" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1 · Basic conditioning\n", "\n", "Given $P(A) = 0.5$, $P(B) = 0.4$ and $P(A \\cap B) = 0.2$, compute $P(A \\mid B)$.\n", "\n", "
Show solution\n", "\n", "$$\n", "P(A \\mid B) = \\frac{0.2}{0.4} = 0.5.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2 · Checking independence\n", "\n", "Are $A$ and $B$ from Exercise 1 independent?\n", "\n", "
Show solution\n", "\n", "Yes. $P(A \\mid B) = 0.5 = P(A)$; equivalently $P(A)P(B) = 0.5 \\times 0.4 = 0.2 = P(A \\cap B)$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 3 · Multiplication rule\n", "\n", "In a trial, $P(T) = 0.5$ and a treated patient recovers with probability $P(R \\mid T) = 0.8$. What is $P(T \\cap R)$?\n", "\n", "
Show solution\n", "\n", "$P(T \\cap R) = P(R \\mid T)P(T) = 0.8 \\times 0.5 = 0.40$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 4 · Law of total probability\n", "\n", "In the same trial, $P(R \\mid T^c) = 0.3$. What is the overall recovery rate $P(R)$?\n", "\n", "
Show solution\n", "\n", "$$\n", "P(R) = P(R \\mid T)P(T) + P(R \\mid T^c)P(T^c) = 0.8(0.5) + 0.3(0.5) = 0.55.\n", "$$\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 5 · Basic Bayes\n", "\n", "A recovered patient is chosen at random. What is the probability they were treated?\n", "\n", "
Show solution\n", "\n", "$$\n", "P(T \\mid R) = \\frac{P(R \\mid T)P(T)}{P(R)} = \\frac{0.40}{0.55} \\approx 0.727.\n", "$$\n", "\n", "
\n", "\n", "## Part 2 · Proofs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 6 · Deriving Bayes' theorem\n", "\n", "Using only the definition of conditional probability, prove Bayes' theorem.\n", "\n", "
Show solution\n", "\n", "1. By definition, $P(A \\mid B) = P(A \\cap B)/P(B)$.\n", "2. Also by definition, $P(B \\mid A) = P(B \\cap A)/P(A)$, so $P(A \\cap B) = P(B \\mid A)P(A)$.\n", "3. Substitute into step 1: $P(A \\mid B) = P(B \\mid A)P(A)/P(B)$.\n", "4. Expanding $P(B)$ by the LTP over $\\{A, A^c\\}$ gives the second form.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 7 · Independence is symmetric\n", "\n", "Show that if $P(A \\mid B) = P(A)$ then $P(B \\mid A) = P(B)$ (assume both probabilities are positive).\n", "\n", "
Show solution\n", "\n", "1. $P(A \\mid B) = P(A)$ means $P(A \\cap B)/P(B) = P(A)$, i.e. $P(A \\cap B) = P(A)P(B)$.\n", "2. Then $P(B \\mid A) = P(A \\cap B)/P(A) = P(A)P(B)/P(A) = P(B)$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 8 · Independence of complements\n", "\n", "Prove that if $A$ and $B$ are independent, so are $A$ and $B^c$.\n", "\n", "
Show solution\n", "\n", "1. Partition $A$: $P(A) = P(A \\cap B) + P(A \\cap B^c)$.\n", "2. So $P(A \\cap B^c) = P(A) - P(A)P(B) = P(A)\\,(1 - P(B)) = P(A)P(B^c)$.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 9 · Conditional probability is a probability\n", "\n", "Show that for fixed $B$ with $P(B) > 0$, $P(\\cdot \\mid B)$ satisfies the normalisation axiom and that $P(A \\mid B) + P(A^c \\mid B) = 1$.\n", "\n", "
Show solution\n", "\n", "1. $P(\\Omega \\mid B) = P(\\Omega \\cap B)/P(B) = P(B)/P(B) = 1$.\n", "2. $A \\cap B$ and $A^c \\cap B$ are disjoint with union $B$, so\n", "\n", "$$\n", "P(A \\mid B) + P(A^c \\mid B) = \\frac{P(A \\cap B) + P(A^c \\cap B)}{P(B)} = \\frac{P(B)}{P(B)} = 1.\n", "$$\n", "\n", "(Non-negativity and additivity carry over the same way, so every rule from Day 1 holds conditionally.)\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 10 · Disjoint is not independent\n", "\n", "Prove that two disjoint events with positive probabilities cannot be independent.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Disjoint means $P(A \\cap B) = 0$, but $P(A)P(B) > 0$.\n", "\n", "**Step 2.** So $P(A \\cap B) \\ne P(A)P(B)$.\n", "\n", "**Step 3.** Intuitively, disjoint events are maximally *dependent*: learning $B$ tells you for certain that $A$ did not happen.\n", "\n", "
\n", "\n", "## Part 3 · Applied" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 11 · Randomisation balances covariates\n", "\n", "In an experiment, 60% of participants are women and every participant is treated with probability 0.5, regardless of sex. What share of the *treated group* is women?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Collect what we know: $P(F) = 0.6$, $P(T \\mid F) = 0.5$, and $P(T) = 0.5$ (everyone has the same chance of treatment).\n", "\n", "**Step 2.** Apply Bayes' theorem:\n", "\n", "$$\n", "P(F \\mid T) = \\frac{P(T \\mid F)\\,P(F)}{P(T)} = \\frac{0.5 \\times 0.6}{0.5} = 0.6.\n", "$$\n", "\n", "**Answer:** 60% of the treated group are women, the same as in the whole sample. Because treatment is independent of sex, the treated group mirrors the population. The same argument holds for any characteristic, including ones nobody measured.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 12 · Screening and the base rate\n", "\n", "A test has sensitivity $P(+ \\mid V) = 0.90$ and false-positive rate $P(+ \\mid V^c) = 0.10$.\n", "\n", "(a) If prevalence is 5%, what is $P(V \\mid +)$?\n", "(b) Repeat with prevalence 30%.\n", "\n", "
Show solution\n", "\n", "**(a) Prevalence 5%**\n", "\n", "**Step 1.** Overall share testing positive (law of total probability):\n", "\n", "$$\n", "P(+) = P(+ \\mid V)P(V) + P(+ \\mid V^c)P(V^c) = 0.90(0.05) + 0.10(0.95) = 0.045 + 0.095 = 0.14.\n", "$$\n", "\n", "**Step 2.** Bayes' theorem:\n", "\n", "$$\n", "P(V \\mid +) = \\frac{0.045}{0.14} \\approx 0.321.\n", "$$\n", "\n", "**(b) Prevalence 30%**\n", "\n", "**Step 3.** $P(+) = 0.90(0.30) + 0.10(0.70) = 0.27 + 0.07 = 0.34$.\n", "\n", "**Step 4.** $P(V \\mid +) = 0.27/0.34 \\approx 0.794$.\n", "\n", "**Answer:** about 32% at 5% prevalence and 79% at 30% prevalence. The same test means very different things in different populations (try it in the Simulation Lab).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 13 · Inflation and interest rates\n", "\n", "Historically $P(I) = 0.40$ (inflation rises), $P(R) = 0.30$ (rates rise) and $P(R \\mid I) = 0.60$. Find $P(I \\cap R)$ and $P(I \\mid R)$. Are the events independent?\n", "\n", "
Show solution\n", "\n", "**Step 1.** Multiplication rule:\n", "\n", "$$\n", "P(I \\cap R) = P(R \\mid I)\\,P(I) = 0.60 \\times 0.40 = 0.24.\n", "$$\n", "\n", "**Step 2.** Definition of conditional probability:\n", "\n", "$$\n", "P(I \\mid R) = \\frac{P(I \\cap R)}{P(R)} = \\frac{0.24}{0.30} = 0.80.\n", "$$\n", "\n", "**Step 3.** Independence check: $P(I)P(R) = 0.40 \\times 0.30 = 0.12 \\ne 0.24$.\n", "\n", "**Answer:** $P(I \\cap R) = 0.24$, $P(I \\mid R) = 0.80$, and the events are not independent: rising rates make rising inflation twice as likely as its unconditional rate.\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 14 · Take-up in an encouragement design\n", "\n", "A subsidy offer $Z$ is randomised 50/50. Take-up is $P(D = 1 \\mid Z = 1) = 0.6$ and $P(D = 1 \\mid Z = 0) = 0.1$.\n", "\n", "(a) What share of the population takes the subsidy?\n", "(b) If nobody is put off by the offer, what share are \"compliers\" (take it only if offered)?\n", "\n", "
Show solution\n", "\n", "**(a) Overall take-up**\n", "\n", "**Step 1.** Law of total probability over the two arms:\n", "\n", "$$\n", "P(D = 1) = P(D = 1 \\mid Z = 1)P(Z = 1) + P(D = 1 \\mid Z = 0)P(Z = 0) = 0.6(0.5) + 0.1(0.5) = 0.35.\n", "$$\n", "\n", "**(b) Share of compliers**\n", "\n", "**Step 2.** People who take the subsidy without an offer (\"always-takers\") make up 10% of the control arm. Randomisation makes the two arms alike, so they are also 10% of the offer arm.\n", "\n", "**Step 3.** If nobody refuses *because* of the offer (no \"defiers\"), the extra take-up in the offer arm comes from compliers:\n", "\n", "$$\n", "P(\\text{complier}) = 0.6 - 0.1 = 0.5.\n", "$$\n", "\n", "**Answer:** 35% take the subsidy; half the population are compliers. This difference in take-up is the *first stage* of an instrumental-variables design (Phase 5).\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 15 · Simpson's paradox with numbers\n", "\n", "Hospital data:\n", "\n", "| | Severe: deaths / patients | Mild: deaths / patients |\n", "|---|---|---|\n", "| Treatment A | 24 / 80 | 1 / 20 |\n", "| Treatment B | 8 / 20 | 8 / 80 |\n", "\n", "Compute mortality for each treatment within each severity level and overall. Explain the reversal with the LTP.\n", "\n", "
Show solution\n", "\n", "**Step 1.** Mortality within each severity level:\n", "\n", "| | Severe | Mild |\n", "|---|---|---|\n", "| A | $24/80 = 30\\%$ | $1/20 = 5\\%$ |\n", "| B | $8/20 = 40\\%$ | $8/80 = 10\\%$ |\n", "\n", "A has lower mortality in both strata.\n", "\n", "**Step 2.** Overall mortality: A $= (24 + 1)/100 = 25\\%$; B $= (8 + 8)/100 = 16\\%$. B looks better.\n", "\n", "**Step 3.** Write each overall rate with the LTP, using each arm's own case mix ($P(S \\mid A) = 0.8$, $P(S \\mid B) = 0.2$):\n", "\n", "$$\n", "P(D \\mid A) = 0.30(0.8) + 0.05(0.2) = 0.25, \\qquad P(D \\mid B) = 0.40(0.2) + 0.10(0.8) = 0.16.\n", "$$\n", "\n", "**Answer:** the reversal happens because A's overall rate puts 80% weight on the deadly severe stratum, while B's puts only 20% there. Severity is a *confounder*; Day 15 shows how to compare the arms fairly.\n", "\n", "
\n", "\n", "## Verify in Python" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*Your work:*" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def bayes(prior, sens, fpr):\n", " p_pos = sens * prior + fpr * (1 - prior)\n", " return sens * prior / p_pos, p_pos\n", "\n", "for prev in (0.05, 0.30):\n", " post, p_pos = bayes(prev, 0.90, 0.10)\n", " print(f\"prevalence {prev:.2f}: P(+) = {p_pos:.3f}, P(V|+) = {post:.3f}\")\n", "\n", "# Simpson's paradox table (Exercise 15)\n", "data = {\"A\": {\"severe\": (24, 80), \"mild\": (1, 20)},\n", " \"B\": {\"severe\": (8, 20), \"mild\": (8, 80)}}\n", "for arm, strata in data.items():\n", " rates = {s: d / n for s, (d, n) in strata.items()}\n", " deaths = sum(d for d, _ in strata.values()); n = sum(n for _, n in strata.values())\n", " print(arm, {k: round(v, 2) for k, v in rates.items()}, \"overall\", deaths / n)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Key takeaways\n", "\n", "- Conditioning shrinks the sample space; independence means conditioning changes nothing.\n", "- Overall rates are weighted averages of subgroup rates (LTP).\n", "- Bayes' theorem turns $P(\\text{evidence} \\mid \\text{hypothesis})$ into $P(\\text{hypothesis} \\mid \\text{evidence})$, and the base rate matters." ] } ] }