Bayes' Theorem: Updating Beliefs with New Evidence

Introduction

In our last lesson, we explored the Law of Large Numbers and the Central Limit Theorem, cornerstones of frequentist statistics that describe the long-run behavior of sample averages. These theorems are powerful, but they treat probabilities as objective, fixed frequencies. Today, we pivot to a different and equally powerful paradigm for reasoning under uncertainty: Bayesian inference.

Instead of viewing probability as a frequency, the Bayesian perspective treats it as a degree of belief. This lesson introduces the engine of this paradigm: Bayes' theorem. You will learn how to use this theorem to formally update your beliefs about the world as you acquire new, incomplete, or noisy evidence. This process of belief updating is fundamental to many advanced AI systems, from robotic navigation and spam filtering to the probabilistic models that underpin generative AI. Our goal is to derive the theorem from first principles and then apply it to see how new data reshapes our understanding.

The Core Idea: Beliefs are Updated, Not Replaced

Before diving into the mathematics, let's build some intuition. A common cognitive bias is to let new evidence completely overshadow what we knew before. Bayesian reasoning provides a principled way to avoid this by combining prior knowledge with new evidence.

The 3Blue1Brown video below uses a classic psychology puzzle to illustrate this beautifully. It shows how failing to account for the "base rate" (the prior probability) can lead to irrational conclusions.

The Language of Bayesian Inference

The intuitive process described in the video has a formal mathematical language. Let's define the components of Bayes' theorem. We'll use the example of a medical diagnosis, which is a common and powerful application. Let $D$ be the event that a patient has a disease, and $T$ be the event that they receive a positive test result.

• Prior $P(D)$: This is your initial belief that the patient has the disease before you see any test results. It's the "base rate" or background probability. For a rare disease, this might be very low.
• Likelihood $P(T|D)$: This is the probability of observing the evidence (a positive test) given that the patient actually has the disease. This is often related to the test's sensitivity.
• Posterior $P(D|T)$: This is the updated belief that the patient has the disease after you have observed the evidence (the positive test). This is what we usually want to compute.
• Evidence (or Marginal Likelihood) $P(T)$: This is the overall probability of observing the evidence (a positive test) under all circumstances, whether the patient has the disease or not.

The following video from Steve Brunton provides a clear walkthrough of this terminology.

A First-Principles Derivation of Bayes' Theorem

Given your interest in first principles, we won't just memorize the formula; we will derive it directly from the definition of conditional probability.

Recall the definition of the conditional probability of event A given event B:

$$P(A|B)=\frac{P(A,B)}{P(B)}$$

where $P(A,B)$ is the joint probability of both A and B occurring.

Now, here is a question for you: How would you write the expression for $P(B|A)$? What component is common to both the expression for $P(A|B)$ and your new expression for $P(B|A)$?

Answer

The expression for $P(B|A)$ is: $$ P(B|A) = \frac{P(A, B)}{P(A)} $$ The common component is the joint probability, $P(A, B)$.

Since the joint probability $P(A,B)$ is the same as $P(B,A)$, we can express it in two ways by rearranging our conditional probability formulas:

1. $P(A,B)=P(A|B)P(B)$
2. $P(A,B)=P(B|A)P(A)$

By setting these two expressions equal to each other, we get:

$$P(A|B)P(B)=P(B|A)P(A)$$

Dividing by $P(B)$ gives us the familiar form of Bayes' theorem:

$$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$$

Using our more descriptive terminology (Posterior, Likelihood, Prior, Evidence):

$$\text{Posterior}=\frac{\text{Likelihood}\times \text{Prior}}{\text{Evidence}}$$

or, for our medical example:

$$P(D|T)=\frac{P(T|D)P(D)}{P(T)}$$

The following resource provides a concise summary of this derivation and related concepts.

Expanding the "Evidence" Term

Often, we don't know the evidence $P(B)$ directly. However, we can calculate it using the Law of Total Probability. An event $B$ can occur either with event $A$ or with event "not $A$" (denoted $A^c$). Therefore, we can break down $P(B)$ as follows:

$$P(B)=P(B,A)+P(B,A^c)$$

Using the definition of conditional probability, this becomes:

$$P(B)=P(B|A)P(A)+P(B|A^c)P(A^c)$$

Substituting this into the denominator of Bayes' theorem gives us its full form, which is often more practical for calculations:

$$P(A|B)=\frac{P(B|A)P(A)}{P(B|A)P(A)+P(B|A^c)P(A^c)}$$

This form is powerful because it allows us to calculate the posterior using only conditional probabilities and priors.

A Powerful Example: The Paradox of Medical Screening

Let's apply the full theorem to a realistic scenario. The cancer screening example from Steve Brunton's lecture is a classic illustration of how Bayesian reasoning can produce counter-intuitive but correct results. It highlights the immense impact of the prior probability, especially for rare events.

From a Single Update to a Recursive Process

The true power of Bayesian inference in computing is that it's not a one-off calculation. It's a continuous process. The posterior belief from one observation becomes the prior belief for the next. This recursive cycle of prediction and updating is the foundation of many algorithms in robotics and AI.

Let's see this in action with a coin toss example. Imagine we have a coin and we want to estimate its probability of landing heads, which we'll call $\theta$. We can start with a prior belief about $\theta$ and then update that belief with each new flip.

The GeeksforGeeks article below provides an excellent walkthrough of this process, including a Python implementation. It also introduces an important concept: conjugate priors, where the prior and posterior distributions belong to the same family, making the math much cleaner. For a coin toss (a Bernoulli process), the Beta distribution is a conjugate prior.

Observing this updating process leads to a key question. As we collect more and more coin flips, the posterior distribution becomes sharper and more concentrated around a single value. How does this Bayesian view of a narrowing posterior distribution relate to the Law of Large Numbers we studied in the last lesson?

Answer

They are two perspectives on the same phenomenon. The Law of Large Numbers states that the sample average $\bar{X}_n$ converges to the true mean $\mu$. In the Bayesian context, as we gather more data, the posterior distribution for the parameter $\mu$ (or $\theta$ in the coin example) collapses to a single point at its true value. The likelihood term, which represents the data, increasingly dominates the prior, and our uncertainty about the true parameter value diminishes. The Bayesian posterior convergence provides a distributional view of the certainty that the frequentist LLN describes as a point-wise convergence.

Conclusion

In this lesson, we shifted our perspective on probability from long-run frequencies to a measure of belief. We saw that Bayes' theorem provides the mathematical framework for updating these beliefs in a principled manner.

Key Takeaways:

• Bayes' Theorem provides a formal way to update a prior belief into a posterior belief after observing new evidence, mediated by the likelihood of that evidence.
• The formula is derived directly from the definition of conditional probability: $P(A|B)=P(B|A)P(A)/P(B)$.
• The denominator, or evidence, can be calculated using the Law of Total Probability, which is essential for practical applications.
• Bayesian inference is a recursive process. The posterior from one step can serve as the prior for the next, allowing systems to learn and adapt as they acquire more data. This is crucial in fields like robotics and online machine learning.

Preview of the Next Lesson:

In the coin toss example, the Bayesian approach gave us a full probability distribution for the coin's bias $\theta$. While this distribution captures our uncertainty, we often need to choose a single best value for $\theta$. How would we do that? One common approach is to pick the value of $\theta$ that makes our observed data most probable. This idea is the foundation of our next topic: Maximum Likelihood Estimation (MLE). We will derive the MLE for a Bernoulli distribution and see how it forms the basis for one of the most common loss functions in deep learning.