Calculating Posterior Probabilities with Bayes' Theorem

1. The Core Idea: Updating Beliefs

Imagine you're trying to guess if a person, Steve, is a librarian or a farmer. You are given a description of his personality: "meek and tidy soul," with a "passion for detail." Most people immediately think "librarian," because the description fits the stereotype.

However, this ignores a crucial piece of information: there are far more farmers than librarians in the population. A rational approach shouldn't just consider how well the evidence fits the hypothesis, but also how likely that hypothesis was in the first place.

Bayes' Theorem is the mathematical rule for combining our initial belief with new evidence. Let's watch a video that explains this beautifully.

The key takeaway is that new evidence doesn't create our belief in a vacuum; it updates a prior belief.

2. The Formula and Its Components

Now let's formalize this process. Bayes' Theorem allows us to calculate $P(H|E)$, the probability of our Hypothesis given the Evidence, using the reverse conditional probability, $P(E|H)$.

The formula is:
$P(H|E)=\frac{P(E|H)\cdot P(H)}{P(E)}$

Let's define each part, using the idea of a medical test for a disease as our hypothesis (H) and a positive test result as our evidence (E).

• Posterior Probability — $P(H|E)$: The probability of the hypothesis after seeing the evidence. This is what we usually want to calculate.

  • Example: The probability you actually have the disease, given that you tested positive.

• Likelihood — $P(E|H)$: The probability of seeing the evidence if the hypothesis is true. This is often a known property of a system (e.g., the accuracy of a test).

  • Example: The probability of testing positive, given that you have the disease (also known as the test's "sensitivity").

• Prior Probability — $P(H)$: The probability of the hypothesis before seeing any evidence. This is our starting belief, often called the "base rate."

  • Example: The overall probability that a random person in the population has the disease.

• Evidence — $P(E)$: The total probability of observing the evidence, regardless of the hypothesis.

  • Example: The overall probability of anyone testing positive.

The denominator, $P(E)$, can be tricky to find directly. We usually calculate it by considering all possible ways the evidence could have occurred. This is known as the Law of Total Probability:
$P(E)=P(E|H)\cdot P(H)+P(E|\neg H)\cdot P(\neg H)$
In plain English: The probability of a positive test is (the chance of a true positive) + (the chance of a false positive).

To see these definitions explained further, let's turn to a great article that connects this directly to machine learning.

3. A Practical Application: Medical Diagnosis

Let's apply the theorem to a realistic problem. This example highlights why Bayes' Theorem is so important and how our intuition can be misleading—a phenomenon known as the base rate fallacy.

Problem: A certain cancer affects 1 in 5000 people. A test for this cancer is 95% accurate (it will be positive for 95% of people with the disease). However, it also has a 6% false positive rate (it will be positive for 6% of people without the disease). If you test positive, what is the probability you actually have cancer?

Our intuition might say the probability is high, around 95%. Let's see what Bayes' Theorem tells us.

Let's first define our terms:

• H = You have cancer.
• E = You test positive.

Now, what are our probabilities?

• Prior, $P(H)$ = 1/5000 = 0.0002
• Likelihood, $P(E|H)$ = 0.95 (test accuracy, or sensitivity)
• Probability of a false positive, $P(E|\neg H)$ = 0.06

First, let's calculate the denominator, $P(E)$, the total probability of testing positive.
$P(E)=P(E|H)\cdot P(H)+P(E|\neg H)\cdot P(\neg H)$
We need $P(\neg H)$, which is simply $1-P(H)=1-0.0002=0.9998$.
$P(E)=(0.95\cdot 0.0002)+(0.06\cdot 0.9998)$
$P(E)=0.00019+0.059988\approx 0.060178$
So, about 6% of all people will test positive.

Now we can calculate the posterior, $P(H|E)$:
$P(H|E)=\frac{P(E|H)\cdot P(H)}{P(E)}=\frac{0.95\cdot 0.0002}{0.060178}\approx 0.00316$

The result is approximately 0.32%. This is shockingly low and goes against our intuition. Even with a positive result from a seemingly accurate test, the probability of having the disease is less than 1%. This is because the disease is so rare (the prior is so low) that a false positive is much more likely than a true positive.

To see this calculation broken down with visual aids, watch the following video. It uses a similar medical example and a tree diagram, which is a fantastic way to visualize the calculations.

As you are aiming to become a Python AI developer, it's also useful to see how this translates into code.

4. Practice Problem

Let's test your understanding. A document from the University of Washington provides several good practice problems. Try this one, which is similar to our medical example.

Problem: The New York State Health Department reports a 10% rate of the HIV virus for the “at-risk” population. A preliminary screening test is correct 95% of the time. If someone from the at-risk population tests positive, what is the probability they have the HIV virus?
(Note: "Correct 95% of the time" implies both a 95% true positive rate and a 95% true negative rate, meaning a 5% false positive rate).

Solution:

• $P(H)=0.10$
• $P(E|H)=0.95$
• $P(E|\neg H)=0.05$ (since the true negative rate is 95%)
• Posterior $P(H|E)\approx 0.678$ or 67.8%.

See if you can arrive at this answer yourself!

Conclusion

Congratulations on completing the "Mathematics for Machine Learning" module! We've covered a lot of ground, from linear algebra to statistics and finally probability theory. Bayes' Theorem is a fitting finale, as it encapsulates a core idea in machine learning: updating what we know based on data.

Key Takeaways:

• Bayes' Theorem is a mathematical formula for updating an initial belief (the prior) with new evidence to get a revised belief (the posterior).
• The formula is $P(H|E)=\frac{P(E|H)\cdot P(H)}{P(E)}$, where $P(E|H)$ is the likelihood.
• It is crucial to account for the prior probability (or base rate) of an event. Ignoring it leads to the base rate fallacy, where we can dramatically overestimate probabilities.
• This theorem is the foundation for Naive Bayes classifiers, Bayesian optimization, and a whole field of statistics called Bayesian inference, all of which are used in AI.

This lesson concludes our journey through the essential mathematics. Now, it's time to get our hands dirty with data. In our next module, "Data Analysis and Visualization," we will dive into pandas, the most important Python library for data manipulation. You'll learn how to load, inspect, clean, and prepare real-world datasets—the very first step in building any machine learning application.