Bayesian Belief Updating in ML

1. The Core Idea: Updating Beliefs with Evidence

At its heart, Bayesian reasoning is a formal process for updating your beliefs in light of new evidence. Imagine you have an initial hypothesis. When you observe something new, how much should that observation change your confidence in your initial hypothesis? Bayes' theorem provides the answer.

This is a concept that people often get wrong intuitively, usually by ignoring the importance of their initial belief, also known as the "prior" or "base rate". The following video from 3Blue1Brown provides a fantastic intuition for this.

As the video explains, the key mantra is that new evidence does not determine your beliefs in a vacuum; it updates prior beliefs.

Let's formalize this. Bayes' theorem is stated as:

$P(H|E)=\frac{P(E|H)\cdot P(H)}{P(E)}$

Let's break down these terms, which you just heard in the video:

• $P(H|E)$: The Posterior Probability. This is what we want to calculate: the probability of our hypothesis $H$ being true, given that we have observed the evidence $E$. It's our updated belief.
• $P(H)$: The Prior Probability. This is our initial belief in the hypothesis $H$ before seeing any evidence. In the video's example, this was the base rate probability of a person being a librarian.
• $P(E|H)$: The Likelihood. This is the probability of observing the evidence $E$ if our hypothesis $H$ were true. In the example, this was the probability that a librarian would fit Steve's description.
• $P(E)$: The Evidence Probability. This is the overall probability of observing the evidence, regardless of the hypothesis. It acts as a normalization constant.

Often, $P(E)$ is calculated by considering all possible hypotheses. For a simple true/false hypothesis, it's expanded using the law of total probability:
$P(E)=P(E|H)P(H)+P(E|\neg H)P(\neg H)$
This is exactly what was done in the video by adding the number of meek farmers and meek librarians to get the total number of meek people.

2. A Practical Example: The Base Rate Fallacy

Our intuition often overvalues the likelihood and ignores the prior, a mistake so common it's called the base rate fallacy. Medical testing is the classic example.

Let's work through a scenario. A certain type of cancer affects 0.5% of the population. A test for this cancer is 97% sensitive (it correctly identifies 97% of people who have the disease) and 95% specific (it correctly identifies 95% of people who don't). If a person tests positive, what is the probability they actually have cancer?

Our intuition, looking at the 97% sensitivity, might say the probability is very high. Let's see what Bayes' theorem says. The following article provides an excellent, practical example.

Let's map our cancer test problem onto the formula:

• $H$: The patient has cancer.
• $E$: The patient tests positive.

We want to find $P(\text{Cancer | Positive})$.

What we know:

• Prior $P(\text{Cancer})=0.005$ (0.5% prevalence)
• Likelihood $P(\text{Positive | Cancer})=0.97$ (97% sensitivity)
• Specificity $P(\text{Negative | No Cancer})=0.95$. From this, we can deduce the false positive rate: $P(\text{Positive | No Cancer})=1-0.95=0.05$.

First, we calculate the total probability of a positive test, $P(\text{Positive})$:
$P(\text{Positive})=P(\text{Positive | Cancer})P(\text{Cancer})+P(\text{Positive | No Cancer})P(\text{No Cancer})$
$P(\text{Positive})=(0.97\cdot 0.005)+(0.05\cdot (1-0.005))$
$P(\text{Positive})=0.00485+0.04975=0.0546$

Now we can apply Bayes' theorem:
$P(\text{Cancer | Positive})=\frac{P(\text{Positive | Cancer})\cdot P(\text{Cancer})}{P(\text{Positive})}$
$P(\text{Cancer | Positive})=\frac{0.97\cdot 0.005}{0.0546}=\frac{0.00485}{0.0546}\approx 0.0888$

So, there is only an 8.9% chance the person actually has cancer, even with a positive test! This counter-intuitive result is driven by the low prior probability. The number of healthy people who get false positives outweighs the number of sick people who get true positives.

3. Bayesian Inference for Machine Learning

Now, let's connect this to machine learning. In Bayesian ML, we treat model parameters as hypotheses. Our goal is to find the most plausible parameters given our data.

• Hypothesis ($H$) becomes our model parameters ($\theta$).
• Evidence ($E$) becomes our observed data ($D$).

The theorem is re-written as:

$\underset{\text{Posterior}}{\underbrace{P(\theta |D)}}\propto \underset{\text{Likelihood}}{\underbrace{P(D|\theta )}}\cdot \underset{\text{Prior}}{\underbrace{P(\theta )}}$

We often use the proportional symbol ($\propto$) because the denominator $P(D)$ is a constant that can be difficult to calculate, but it doesn't change the relative probabilities of different parameter values.

This framework gives us a way to find a full probability distribution for our model's parameters, which represents our belief (and uncertainty!) about the parameters after seeing the data. The following video provides an exceptionally clear, practical demonstration of this process.

The process you just saw is a beautiful illustration of Bayesian inference in action. We started with a "flat" prior (total uncertainty) and ended with a posterior distribution peaked around 38%, which represents our updated, data-informed belief about the true signup rate.

4. Sequential Updates: Learning Over Time

One of the most powerful aspects of this framework is its ability to learn sequentially. If we get more data, we don't need to start from scratch. The posterior from our first analysis simply becomes the prior for our next one.

The article on drug screening you looked at earlier has an excellent section that demonstrates this.

This "chaining" process is exactly what allows a machine learning model to update its internal parameters as it sees more and more batches of data during training. Each batch of data sharpens the model's "beliefs" about the world.

Conclusion

Today we've unpacked one of the most important ideas in statistics. You've seen how Bayes' theorem provides a rational way to update our beliefs and how this same principle is applied in machine learning to allow models to learn from data.

Key Takeaways:

• Bayes' theorem ($P(H|E)\propto P(E|H)P(H)$) is a rule for updating a prior belief into a posterior belief after observing evidence.
• Ignoring the prior (or base rate) can lead to the base rate fallacy, where we make intuitive but incorrect judgments about probability.
• In machine learning, we use Bayesian inference to find the posterior distribution of model parameters given data: $P(\theta |D)\propto P(D|\theta )P(\theta )$.
• This posterior represents the model's learned knowledge and its uncertainty about the parameters.
• The process is naturally sequential: the posterior after one set of evidence becomes the prior for the next, allowing models to learn iteratively.

Next Steps:
In this lesson, our output was a full probability distribution (the posterior). To make practical use of such distributions, we need ways to summarize them. In our next lesson, we will explore two of the most important summary statistics: expected value, which gives us a single "best guess" from a distribution, and variance, which quantifies the uncertainty or spread around that guess.