Day 2: Conditional Probability, Independence & Bayes

Theory

Conditioning restricts the sample space

Causal questions are almost never about raw probabilities. We want the probability of employment given training, or of recovery given a drug. Learning that $B$ happened shrinks the world to $B$ :

P(A \mid B) = \frac{P(A \cap B)}{P(B)}, \qquad P(B) > 0.

Rearranged, this is the multiplication rule: $P(A \cap B) = P(A \mid B)\,P(B)$ .

Independence

$A$ and $B$ are independent if

P(A \cap B) = P(A)\,P(B) \quad\Longleftrightarrow\quad P(A \mid B) = P(A) \;\;(\text{when } P(B) > 0).

Knowing $B$ tells you nothing about $A$ . Randomisation makes treatment independent of every pre-treatment characteristic, observed or not. That single fact is why experiments identify causal effects.

Law of total probability (LTP)

If $B_1, \dots, B_k$ partition $\Omega$ :

P(A) = \sum_{j=1}^{k} P(A \mid B_j)\,P(B_j).

An overall rate is a weighted average of subgroup rates, with weights equal to subgroup shares. Keep this in mind: when two groups have different weights, their overall rates can mislead.

Bayes' theorem

P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)} = \frac{P(B \mid A)\,P(A)}{P(B \mid A)P(A) + P(B \mid A^c)P(A^c)}.

Bayes reverses the direction of conditioning: from "how likely is the evidence given the hypothesis" to "how likely is the hypothesis given the evidence". The base rate $P(A)$ matters enormously.

Causal link. "Treatment is independent of potential outcomes" is the key identifying assumption of an experiment. Today's definition of independence is what that sentence means.

Part 1 · Warm-up

Exercise 1 · Basic conditioning

Given $P(A) = 0.5$ , $P(B) = 0.4$ and $P(A \cap B) = 0.2$ , compute $P(A \mid B)$ .

Show solution

P(A \mid B) = \frac{0.2}{0.4} = 0.5.

Exercise 2 · Checking independence

Are $A$ and $B$ from Exercise 1 independent?

Show solution

Yes. $P(A \mid B) = 0.5 = P(A)$ ; equivalently $P(A)P(B) = 0.5 \times 0.4 = 0.2 = P(A \cap B)$ .

Exercise 3 · Multiplication rule

In a trial, $P(T) = 0.5$ and a treated patient recovers with probability $P(R \mid T) = 0.8$ . What is $P(T \cap R)$ ?

Show solution

$P(T \cap R) = P(R \mid T)P(T) = 0.8 \times 0.5 = 0.40$ .

Exercise 4 · Law of total probability

In the same trial, $P(R \mid T^c) = 0.3$ . What is the overall recovery rate $P(R)$ ?

Show solution

P(R) = P(R \mid T)P(T) + P(R \mid T^c)P(T^c) = 0.8(0.5) + 0.3(0.5) = 0.55.

Exercise 5 · Basic Bayes

A recovered patient is chosen at random. What is the probability they were treated?

Show solution

P(T \mid R) = \frac{P(R \mid T)P(T)}{P(R)} = \frac{0.40}{0.55} \approx 0.727.

Part 2 · Proofs

Exercise 6 · Deriving Bayes' theorem

Using only the definition of conditional probability, prove Bayes' theorem.

Show solution

By definition, $P(A \mid B) = P(A \cap B)/P(B)$ .
Also by definition, $P(B \mid A) = P(B \cap A)/P(A)$ , so $P(A \cap B) = P(B \mid A)P(A)$ .
Substitute into step 1: $P(A \mid B) = P(B \mid A)P(A)/P(B)$ .
Expanding $P(B)$ by the LTP over $\{A, A^c\}$ gives the second form.

Exercise 7 · Independence is symmetric

Show that if $P(A \mid B) = P(A)$ then $P(B \mid A) = P(B)$ (assume both probabilities are positive).

Show solution

$P(A \mid B) = P(A)$ means $P(A \cap B)/P(B) = P(A)$ , i.e. $P(A \cap B) = P(A)P(B)$ .
Then $P(B \mid A) = P(A \cap B)/P(A) = P(A)P(B)/P(A) = P(B)$ .

Exercise 8 · Independence of complements

Prove that if $A$ and $B$ are independent, so are $A$ and $B^c$ .

Show solution

Partition $A$ : $P(A) = P(A \cap B) + P(A \cap B^c)$ .
So $P(A \cap B^c) = P(A) - P(A)P(B) = P(A)\,(1 - P(B)) = P(A)P(B^c)$ .

Exercise 9 · Conditional probability is a probability

Show that for fixed $B$ with $P(B) > 0$ , $P(\cdot \mid B)$ satisfies the normalisation axiom and that $P(A \mid B) + P(A^c \mid B) = 1$ .

Show solution

$P(\Omega \mid B) = P(\Omega \cap B)/P(B) = P(B)/P(B) = 1$ .
$A \cap B$ and $A^c \cap B$ are disjoint with union $B$ , so

P(A \mid B) + P(A^c \mid B) = \frac{P(A \cap B) + P(A^c \cap B)}{P(B)} = \frac{P(B)}{P(B)} = 1.

(Non-negativity and additivity carry over the same way, so every rule from Day 1 holds conditionally.)

Exercise 10 · Disjoint is not independent

Prove that two disjoint events with positive probabilities cannot be independent.

Show solution

Step 1. Disjoint means $P(A \cap B) = 0$ , but $P(A)P(B) > 0$ .

Step 2. So $P(A \cap B) \ne P(A)P(B)$ .

Step 3. Intuitively, disjoint events are maximally dependent: learning $B$ tells you for certain that $A$ did not happen.

Part 3 · Applied

Exercise 11 · Randomisation balances covariates

In an experiment, 60% of participants are women and every participant is treated with probability 0.5, regardless of sex. What share of the treated group is women?

Show solution

Step 1. Collect what we know: $P(F) = 0.6$ , $P(T \mid F) = 0.5$ , and $P(T) = 0.5$ (everyone has the same chance of treatment).

Step 2. Apply Bayes' theorem:

P(F \mid T) = \frac{P(T \mid F)\,P(F)}{P(T)} = \frac{0.5 \times 0.6}{0.5} = 0.6.

Answer: 60% of the treated group are women, the same as in the whole sample. Because treatment is independent of sex, the treated group mirrors the population. The same argument holds for any characteristic, including ones nobody measured.

Exercise 12 · Screening and the base rate

A test has sensitivity $P(+ \mid V) = 0.90$ and false-positive rate $P(+ \mid V^c) = 0.10$ .

(a) If prevalence is 5%, what is $P(V \mid +)$ ? (b) Repeat with prevalence 30%.

Show solution

(a) Prevalence 5%

Step 1. Overall share testing positive (law of total probability):

P(+) = P(+ \mid V)P(V) + P(+ \mid V^c)P(V^c) = 0.90(0.05) + 0.10(0.95) = 0.045 + 0.095 = 0.14.

Step 2. Bayes' theorem:

P(V \mid +) = \frac{0.045}{0.14} \approx 0.321.

(b) Prevalence 30%

Step 3. $P(+) = 0.90(0.30) + 0.10(0.70) = 0.27 + 0.07 = 0.34$ .

Step 4. $P(V \mid +) = 0.27/0.34 \approx 0.794$ .

Answer: about 32% at 5% prevalence and 79% at 30% prevalence. The same test means very different things in different populations (try it in the Simulation Lab).

Exercise 13 · Inflation and interest rates

Historically $P(I) = 0.40$ (inflation rises), $P(R) = 0.30$ (rates rise) and $P(R \mid I) = 0.60$ . Find $P(I \cap R)$ and $P(I \mid R)$ . Are the events independent?

Show solution

Step 1. Multiplication rule:

P(I \cap R) = P(R \mid I)\,P(I) = 0.60 \times 0.40 = 0.24.

Step 2. Definition of conditional probability:

P(I \mid R) = \frac{P(I \cap R)}{P(R)} = \frac{0.24}{0.30} = 0.80.

Step 3. Independence check: $P(I)P(R) = 0.40 \times 0.30 = 0.12 \ne 0.24$ .

Answer: $P(I \cap R) = 0.24$ , $P(I \mid R) = 0.80$ , and the events are not independent: rising rates make rising inflation twice as likely as its unconditional rate.

Exercise 14 · Take-up in an encouragement design

A subsidy offer $Z$ is randomised 50/50. Take-up is $P(D = 1 \mid Z = 1) = 0.6$ and $P(D = 1 \mid Z = 0) = 0.1$ .

(a) What share of the population takes the subsidy? (b) If nobody is put off by the offer, what share are "compliers" (take it only if offered)?

Show solution

(a) Overall take-up

Step 1. Law of total probability over the two arms:

P(D = 1) = P(D = 1 \mid Z = 1)P(Z = 1) + P(D = 1 \mid Z = 0)P(Z = 0) = 0.6(0.5) + 0.1(0.5) = 0.35.

(b) Share of compliers

Step 2. People who take the subsidy without an offer ("always-takers") make up 10% of the control arm. Randomisation makes the two arms alike, so they are also 10% of the offer arm.

Step 3. If nobody refuses because of the offer (no "defiers"), the extra take-up in the offer arm comes from compliers:

P(\text{complier}) = 0.6 - 0.1 = 0.5.

Answer: 35% take the subsidy; half the population are compliers. This difference in take-up is the first stage of an instrumental-variables design (Phase 5).

Exercise 15 · Simpson's paradox with numbers

Hospital data:

	Severe: deaths / patients	Mild: deaths / patients
Treatment A	24 / 80	1 / 20
Treatment B	8 / 20	8 / 80

Compute mortality for each treatment within each severity level and overall. Explain the reversal with the LTP.

Show solution

Step 1. Mortality within each severity level:

	Severe	Mild
A	$24/80 = 30\%$	$1/20 = 5\%$
B	$8/20 = 40\%$	$8/80 = 10\%$

A has lower mortality in both strata.

Step 2. Overall mortality: A $= (24 + 1)/100 = 25\%$ ; B $= (8 + 8)/100 = 16\%$ . B looks better.

Step 3. Write each overall rate with the LTP, using each arm's own case mix ( $P(S \mid A) = 0.8$ , $P(S \mid B) = 0.2$ ):

P(D \mid A) = 0.30(0.8) + 0.05(0.2) = 0.25, \qquad P(D \mid B) = 0.40(0.2) + 0.10(0.8) = 0.16.

Answer: the reversal happens because A's overall rate puts 80% weight on the deadly severe stratum, while B's puts only 20% there. Severity is a confounder; Day 15 shows how to compare the arms fairly.

Verify in Python

def bayes(prior, sens, fpr):
    p_pos = sens * prior + fpr * (1 - prior)
    return sens * prior / p_pos, p_pos

for prev in (0.05, 0.30):
    post, p_pos = bayes(prev, 0.90, 0.10)
    print(f"prevalence {prev:.2f}: P(+) = {p_pos:.3f}, P(V|+) = {post:.3f}")

# Simpson's paradox table (Exercise 15)
data = {"A": {"severe": (24, 80), "mild": (1, 20)},
        "B": {"severe": (8, 20), "mild": (8, 80)}}
for arm, strata in data.items():
    rates = {s: d / n for s, (d, n) in strata.items()}
    deaths = sum(d for d, _ in strata.values()); n = sum(n for _, n in strata.values())
    print(arm, {k: round(v, 2) for k, v in rates.items()}, "overall", deaths / n)

Key takeaways

Conditioning shrinks the sample space; independence means conditioning changes nothing.
Overall rates are weighted averages of subgroup rates (LTP).
Bayes' theorem turns $P(\text{evidence} \mid \text{hypothesis})$ into $P(\text{hypothesis} \mid \text{evidence})$ , and the base rate matters.

Day 2. Conditional Probability, Independence & Bayes

Theory

Conditioning restricts the sample space

Independence

Law of total probability (LTP)

Bayes' theorem

Part 1 · Warm-up

Exercise 1 · Basic conditioning

Exercise 2 · Checking independence

Exercise 3 · Multiplication rule

Exercise 4 · Law of total probability

Exercise 5 · Basic Bayes

Part 2 · Proofs

Exercise 6 · Deriving Bayes' theorem

Exercise 7 · Independence is symmetric

Exercise 8 · Independence of complements

Exercise 9 · Conditional probability is a probability

Exercise 10 · Disjoint is not independent

Part 3 · Applied

Exercise 11 · Randomisation balances covariates

Exercise 12 · Screening and the base rate

Exercise 13 · Inflation and interest rates

Exercise 14 · Take-up in an encouragement design

Exercise 15 · Simpson's paradox with numbers

Verify in Python

Key takeaways

Simulation Lab: Bayes' Theorem Calculator