Bayes' Theorem: Updating Beliefs

Recommended resources

The freely available resources this lesson will direct you to.

Hello! Welcome to your next lesson in our journey through the Foundations of Probability Theory.

In our last lesson, we established the rules of conditional probability and multiplication, seeing how $P (A \cap B) = P (A ∣ B) P (B)$ allows us to analyze sequential events. We concluded with a preview that these simple rules are all we need to unlock one of the most powerful tools in statistics and machine learning.

Today, we will derive and apply that tool: Bayes' Theorem. Your learning outcome is to be able to derive and apply Bayes' theorem for updating beliefs. This theorem provides a formal mechanism for reasoning under uncertainty and quantitatively updating our knowledge as we gather new evidence. This principle is not just a theoretical curiosity; it's the mathematical engine behind many modern AI systems that learn from data.

1. Derivation and Components of Bayes' Theorem

The derivation of Bayes' theorem is surprisingly straightforward, flowing directly from the definition of conditional probability you've already mastered.

Recall the multiplication rule can be written in two ways:

$P (A \cap B) = P (A ∣ B) P (B)$
$P (B \cap A) = P (B ∣ A) P (A)$

Since the intersection is commutative ( $A \cap B = B \cap A$ ), we can set the right-hand sides of these equations equal to each other:
$P (B ∣ A) P (A) = P (A ∣ B) P (B)$

Now, if we are interested in finding $P (B ∣ A)$ and assume $P (A) > 0$ , we can simply divide by $P (A)$ to isolate it:
$P (B ∣ A) = \frac{P ( A ∣ B ) P ( B )}{P ( A )}$

This is Bayes' theorem. To make its application in inference clearer, it's common to use $H$ for a hypothesis and $E$ for evidence. In this notation, the theorem is:
$P (H ∣ E) = \frac{P ( E ∣ H ) P ( H )}{P ( E )}$

This equation provides a way to "invert" the conditional probability. Often, we know $P (E ∣ H)$ (e.g., the probability of a symptom given a disease), but what we really want to know is $P (H ∣ E)$ (the probability of the disease given the symptom).

To explore the derivation and the meaning of each component, please watch the first part of the following video.

Bayes' Theorem (with Example!)

This video by Steve Brunton provides a concise derivation of Bayes' Theorem and clearly explains the terminology used in Bayesian inference.

Watch from the beginning to 05:45. Focus on how the theorem is just a rearrangement of the multiplication rule and understand the names given to each term: prior, posterior, and likelihood.

As the video explained, each term in Bayes' theorem has a special name that reflects its role in the process of updating beliefs:

Bayes' Theorem Formula with Annotations — This diagram illustrates the components of Bayes' theorem. The **prior** is our initial belief, the **likelihood** is how probable the evidence is under our hypothesis, and the **posterior** is our updated belief after seeing the evidence.

Posterior Probability, $P (H ∣ E)$ : The probability of the hypothesis after observing the evidence. This is our updated belief.
Likelihood, $P (E ∣ H)$ : The probability of observing the evidence given that the hypothesis is true. This quantifies how well the hypothesis explains the evidence.
Prior Probability, $P (H)$ : The probability of the hypothesis before observing the evidence. This is our initial belief.
Evidence, $P (E)$ : The total probability of observing the evidence. This acts as a normalization constant, ensuring the posterior probabilities over all possible hypotheses sum to 1.

The core idea can be stated elegantly as:
$posterior \propto likelihood \times prior$

2. Expanding the Evidence: The Law of Total Probability

The denominator, $P (E)$ , can be calculated using the law of total probability, which we've touched upon before. If we have a set of mutually exclusive and exhaustive hypotheses ${H_{1}, H_{2}, \dots, H_{n}}$ that partition the sample space, then the total probability of the evidence $E$ is the sum of its probabilities under each hypothesis, weighted by the prior probability of each hypothesis:
$P (E) = \sum_{i = 1}^{n} P (E \cap H_{i}) = \sum_{i = 1}^{n} P (E ∣ H_{i}) P (H_{i})$
For the simple case of a single hypothesis $H$ and its complement $H^{c}$ , the law of total probability gives:
$P (E) = P (E ∣ H) P (H) + P (E ∣ H^{c}) P (H^{c})$
Substituting this into Bayes' theorem gives its expanded form:
$P (H ∣ E) = \frac{P ( E ∣ H ) P ( H )}{P ( E ∣ H ) P ( H ) + P ( E ∣ H ^{c} ) P ( H ^{c} )}$
This form is particularly useful when you don't know $P (E)$ directly but can figure out the likelihood of the evidence under all possible scenarios.

3. Systematic Bayesian Updating: The Tabular Method

When dealing with multiple competing hypotheses, calculating the posterior for each one individually can be cumbersome. The Bayesian update table provides a systematic and organized way to compute all posterior probabilities at once.

The following video from an MIT course introduces this powerful method.

Sample Class: Class 12--Bayesian Updating: Discrete Priors

In this segment, the instructor introduces a clear, tabular method for performing Bayesian updating. This is a fundamental technique for applying Bayes' theorem in practice.

Watch from 7:24 to 14:15. Pay close attention to the structure of the table: Hypothesis, Prior, Likelihood, Bayes Numerator, and Posterior. Understand how each column is calculated from the previous ones.

Let's formalize the structure of the Bayesian update table you just saw:

| Hypothesis ( $H_{i}$ ) | Prior $P (H_{i})$ | Likelihood $P (E ∣ H_{i})$ | Bayes Numerator $P (E ∣ H_{i}) P (H_{i})$ | Posterior $P (H_{i} ∣ E)$ |
| :--- | :--- | :--- | :--- | :--- |
| $H_{1}$ | $P (H_{1})$ | $P (E ∣ H_{1})$ | $P (E ∣ H_{1}) P (H_{1})$ | $P (H_{1} ∣ E)$ |
| $H_{2}$ | $P (H_{2})$ | $P (E ∣ H_{2})$ | $P (E ∣ H_{2}) P (H_{2})$ | $P (H_{2} ∣ E)$ |
| ... | ... | ... | ... | ... |
| $H_{n}$ | $P (H_{n})$ | $P (E ∣ H_{n})$ | $P (E ∣ H_{n}) P (H_{n})$ | $P (H_{n} ∣ E)$ |
| Total | 1 | (Does not sum to 1) | P(E) | 1 |

Steps to use the table:

Hypotheses: List all mutually exclusive and exhaustive hypotheses.
Priors: Assign a prior probability to each hypothesis. This column must sum to 1.
Likelihoods: For the specific evidence $E$ you observed, calculate the likelihood of that evidence under each hypothesis. This column does not need to sum to 1.
Bayes Numerator: Multiply the Prior by the Likelihood for each row.
Normalize: Sum the "Bayes Numerator" column. This sum is the total probability of the evidence, $P (E)$ .
Posteriors: Divide the "Bayes Numerator" value in each row by the sum $P (E)$ to get the posterior probability. This final column must sum to 1.

Test your understanding!

Let's apply this tabular method to the classic cancer screening problem. A rare disease affects 1 in 1000 people (0.1%). A test for this disease is 99% accurate: it has a 99% true positive rate and a 99% true negative rate (implying a 1% false positive rate). A person is chosen at random and tests positive. What is the probability they actually have the disease?

Construct a Bayesian update table to find the answer.

Hypotheses: $H_{D}$ (has disease), $H_{N D}$ (no disease).
Evidence: $E_{p os}$ (tests positive).

Show answer

First, let's identify our probabilities:

Prior $P (H_{D}) = 0.001$ . Thus, $P (H_{N D}) = 1 - 0.001 = 0.999$ .
Likelihood $P (E_{p os} ∣ H_{D}) = 0.99$ (true positive rate).
Likelihood $P (E_{p os} ∣ H_{N D}) = 0.01$ (false positive rate, since true negative is 99%).

Now we build the table:

| Hypothesis ( $H_{i}$ ) | Prior $P (H_{i})$ | Likelihood $P (E_{p os} ∣ H_{i})$ | Bayes Numerator $P (E_{p os} ∣ H_{i}) P (H_{i})$ | Posterior $P (H_{i} ∣ E_{p os})$ |
| :--- | :--- | :--- | :--- | :--- |
| $H_{D}$ | 0.001 | 0.99 | $0.001 \times 0.99 = 0.00099$ | $0.00099/0.01098 \approx 0.090$ |
| $H_{N D}$ | 0.999 | 0.01 | $0.999 \times 0.01 = 0.00999$ | $0.00999/0.01098 \approx 0.910$ |
| Total | 1 | | P(E) = 0.01098 | 1 |

The posterior probability of having the disease given a positive test, $P (H_{D} ∣ E_{p os})$ , is approximately 9%. This counter-intuitive result, known as the base rate fallacy, highlights the immense impact of a low prior probability (a rare disease) even with a highly accurate test.

4. Sequential Updating: Learning Over Time

The true power of Bayesian inference emerges when we receive multiple pieces of evidence. The process is naturally iterative.

Sample Class: Class 12--Bayesian Updating: Discrete Priors

This final clip explains the process of sequential updating, where the posterior from one step becomes the prior for the next. This is the essence of Bayesian learning.

Watch from 30:52 to 36:34. Focus on the core idea that you don't need to start from scratch with new data. Your current state of belief (the posterior) is simply the starting point (the new prior) for incorporating the next piece of evidence.

The key principle of sequential updating is:
The posterior distribution after one piece of evidence becomes the prior distribution for the next piece of evidence.

This elegant property means you can continually refine your beliefs as more data becomes available. This is fundamental to how many machine learning systems work, from spam filters that learn from each email you mark as spam, to complex models that update their parameters as they process a stream of data.

Conclusion

In this lesson, you have derived Bayes' theorem and learned how to apply it as a formal engine for updating beliefs. We've seen that this is not just an abstract formula, but a practical method for reasoning and learning in the face of uncertainty.

Key Takeaways:

Bayes' Theorem, $P (H ∣ E) = \frac{P ( E ∣ H ) P ( H )}{P ( E )}$ , is a direct consequence of the definition of conditional probability.
It allows us to "invert" probabilities, finding $P (H ∣ E)$ from $P (E ∣ H)$ .
The relationship posterior ∝ likelihood × prior captures the essence of updating beliefs.
The Bayesian update table is a systematic tool for calculating posterior probabilities across multiple competing hypotheses.
Bayesian inference is an iterative process: the posterior after observing data becomes the new prior for subsequent observations.

Preview of the Next Lesson:
So far, our events and hypotheses have been discrete (e.g., coin types, disease status). To apply these ideas to the continuous parameters found in most machine learning models (like the weights in a regression model), we need to generalize from events to random variables. In our next lesson, we will define random variables, distinguish between discrete and continuous types, and introduce the concept of the Probability Mass Function (PMF) to describe the probabilities associated with discrete random variables.