Bayesian Classification: Updating Beliefs with Data

Recommended resources

The freely available resources this lesson will direct you to.

Welcome back to our course on the theoretical foundations of machine learning. In our previous lesson, we established a probabilistic vocabulary, describing how distributions like Bernoulli, Categorical, and Gaussian can serve as mathematical models for the data we observe. We are now equipped to describe the "shape" of data, both for class labels and continuous features.

This lesson takes the next logical step. If we have a model of how data is generated, how do we reason in reverse? That is, given a piece of data we have observed, how can we update our beliefs about the underlying process that generated it? This question is at the heart of learning from data. We will explore Bayes' theorem, the fundamental mathematical rule for updating beliefs in the face of new evidence. You will learn to apply this theorem to build a foundational understanding of probabilistic classification, a cornerstone of machine learning.

The Core Idea: Updating Beliefs

Before diving into equations, it's crucial to grasp the intuition. Bayes' theorem is a formalization of a very common-sense idea: new evidence should update our prior beliefs, not replace them entirely. A famous thought experiment illustrates this beautifully.

Let's watch a segment from the 3Blue1Brown channel that introduces this idea and provides a powerful geometric intuition for Bayes' theorem.

Bayes theorem, the geometry of changing beliefs

This video by Grant Sanderson provides an exceptionally clear and intuitive explanation of the core reasoning behind Bayes' theorem.

Please watch from the introduction of the "Steve the librarian" problem up to the point where the general reasoning is complete. Pay close attention to how the initial ratio of farmers to librarians (the prior) is a crucial piece of information that is often ignored. Next, continue watching the section that formalizes this reasoning into the equation of Bayes' theorem. Notice how each term in the formula corresponds directly to a number used in the example. Finally, watch the segment on the geometric interpretation, which visualizes probabilities as areas. Given your background, this representation of "restricting the space of possibilities" should be particularly clarifying.

The key insight from the video is that the probability of a hypothesis being true after seeing evidence depends on three things:

How likely the hypothesis was in the first place (the prior).
How likely it was to see that evidence if the hypothesis were true (the likelihood).
How likely it was to see that evidence in general.

The Mechanics of Bayesian Inference

Let's formalize what we've just seen. In a machine learning context, our "hypothesis" $H$ is typically a statement about which class a data point belongs to (e.g., $y = spam$ ), and our "evidence" $E$ is the data point itself, represented by its feature vector $x$ .

Bayes' theorem is stated as:

P (y ∣ x) = \frac{P ( x ∣ y ) P ( y )}{P ( x )}

The terms are:

Posterior Probability $P (y ∣ x)$ : The probability of class $y$ given the observed data $x$ . This is what we want to compute. It represents our updated belief.
Likelihood $P (x ∣ y)$ : The probability of observing data $x$ if it belonged to class $y$ . This is where the distributions we learned in the last lesson come in. We might model $P (x ∣ y = spam)$ with a particular distribution of words.
Prior Probability $P (y)$ : The probability of class $y$ before seeing any data. This captures our initial belief, such as the overall proportion of spam emails in the world.
Evidence $P (x)$ : The total probability of observing data $x$ , across all possible classes. It acts as a normalization constant, ensuring that the posterior probabilities sum to 1. It is calculated by summing over all classes: $P (x) = \sum_{y^{'}} P (x ∣ y^{'}) P (y^{'})$ .

The following reading provides a concise, formal procedure for Bayesian inference.

Chapter 12 Bayesian Inference

This document from Carnegie Mellon University's "Statistical Machine Learning" course lays out the formal steps of Bayesian inference.

In Section 12.2, "Basic Concepts," read the subsection "The Mechanics of Bayesian Inference". This section codifies the intuitive process into a three-step procedure and presents the formula with its standard components: prior, model (likelihood), and posterior.

Application: The Bayes Optimal Classifier

The power of Bayes' theorem in classification comes from this simple, yet profound, idea: to make the best possible guess, we should choose the class that is most probable given the data. In other words, the optimal decision is to choose the class $y$ that maximizes the posterior probability $P (y ∣ x)$ .

This gives us the Bayes Optimal Decision Rule:

predicted class = y ar g max P (y ∣ x) = y ar g max \frac{P ( x ∣ y ) P ( y )}{P ( x )}

Since $P (x)$ is the same for all classes $y$ , it doesn't affect the argmax. We can therefore simplify the decision rule to:

predicted class = y ar g max P (x ∣ y) P (y)

This is extremely powerful. It tells us that to build an optimal classifier, we need two things: the class priors $P (y)$ and the class-conditional likelihoods $P (x ∣ y)$ .

The following video provides an excellent walkthrough of this concept using a practical example.

Machine Learning: Bayes Decision Theory

This video illustrates the concept of Bayes Decision Theory for classification.

Watch the video from the beginning until the end of the discussion on the optimal decision function. The video will walk you through: Setting up a classification problem (salmon vs. catfish). Defining the prior P(\omega_j) and the likelihood P(x|\omega_j). Formulating the optimal decision rule as maximizing the posterior. Showing the simplification by removing P(x) and applying the logarithm.

A Concrete Example: Updating Probabilities

Let's solidify this with a numerical example. Imagine an oil exploration scenario. Based on historical data, the prior probability of discovering oil in a new location is 10%, so $P (Oil) = 0.1$ . A common geological indicator is a "bright spot" on a seismic scan.

If there is oil, there is a 90% chance of seeing a bright spot. This is our likelihood: $P (Bright Spot ∣ Oil) = 0.9$ .
If there is no oil, there is still a 20% chance of seeing a bright spot (due to other geological features). This is $P (Bright Spot ∣ No Oil) = 0.2$ .

Now, suppose we perform a scan and find a bright spot. What is our new, updated probability of finding oil? We want to calculate $P (Oil ∣ Bright Spot)$ .

Using Bayes' theorem:

P (Oil ∣ Bright Spot) = \frac{P ( Bright Spot ∣ Oil ) P ( Oil )}{P ( Bright Spot )}

First, we need the denominator, the total probability of seeing a bright spot:

P (Bright Spot) = P (Bright Spot ∣ Oil) P (Oil) + P (Bright Spot ∣ No Oil) P (No Oil) = (0.9 \times 0.1) + (0.2 \times 0.9) = 0.09 + 0.18 = 0.27

Now we can calculate the posterior:

P (Oil ∣ Bright Spot) = \frac{0.09}{0.27} \approx 0.333 or 33.3%

Our belief in finding oil has increased from a 10% prior to a 33.3% posterior after observing the evidence of a bright spot.

This decision tree illustrates the oil discovery problem. The top half of the diagram shows the forward probabilities, starting from the prior and branching based on the likelihoods. The bottom half shows the reverse, or Bayesian, inference, calculating the posterior probability of oil given the evidence of a bright spot.

From Bayesian Theory to ML Practice

A fascinating aspect of this framework is how it connects directly to the terminology and metrics you will encounter constantly in machine learning. Concepts like Precision, Recall, and AUC are not arbitrary; they are deeply rooted in Bayesian reasoning.

Classifiers are easy, if you think Bayes! | Towards Data Science

This article provides an insightful bridge between the abstract language of Bayes' theorem and the practical language of classifier evaluation.

Read the entire article, from start to finish. Focus on understanding the direct mapping the author makes between: Prior Odds and Class Prevalence (imbalance). Likelihood Ratio and TPR/FPR (which relates to ROC/AUC). Posterior Odds and Precision.

This perspective is incredibly useful. It clarifies, for instance, why AUC is considered a measure of a classifier's intrinsic performance (it's related to the likelihood ratio, independent of the prior), while Precision is sensitive to class imbalance (as it represents the posterior, which depends on the prior).

Case Study: The Naive Bayes Classifier

The most direct application of Bayes' theorem in classification is the Naive Bayes algorithm. It computes the posterior probability for each class and makes its prediction based on the Bayes optimal decision rule.

Its "naive" quality comes from a simplifying assumption: it assumes that all features $x_{i}$ are conditionally independent of each other, given the class $y$ . This allows us to write the likelihood $P (x ∣ y)$ as a simple product:

P (x_{1}, \dots, x_{d} ∣ y) = P (x_{1} ∣ y) \times P (x_{2} ∣ y) \times \dots \times P (x_{d} ∣ y) = i = 1 \prod d P (x_{i} ∣ y)

Even though this independence assumption is often violated in reality, the classifier works surprisingly well in many applications, such as text classification and spam filtering.

This infographic provides a step-by-step walkthrough of the Multinomial Naive Bayes algorithm, a variant commonly used for text classification. It clearly shows the progression from Bayes' theorem (Step 1) to a prediction template (Step 2), applying the multinomial likelihood model (Step 3 & 4), estimating parameters from a training set (Step 5), and finally predicting the class of a new sample (Step 6).

This model encapsulates today's lesson perfectly: it takes the abstract formula of Bayes' theorem and turns it into a concrete, trainable classification algorithm by defining a specific model for the likelihood.

Conclusion

In this lesson, you've moved from simply modeling data to reasoning about it. We've seen how Bayes' theorem provides a principled, mathematical framework for updating our beliefs in light of new evidence.

Key takeaways from this lesson are:

Bayes' theorem $P (y ∣ x) = \frac{P ( x ∣ y ) P ( y )}{P ( x )}$ is the fundamental rule for updating a prior belief $P (y)$ into a posterior belief $P (y ∣ x)$ after observing evidence $x$ .
The Bayes optimal classifier makes predictions by choosing the class with the highest posterior probability, effectively implementing the rule $ar g max_{y} P (x ∣ y) P (y)$ .
The components of Bayesian inference have direct parallels in machine learning evaluation: priors relate to class imbalance, likelihoods relate to metrics like AUC, and posteriors relate to metrics like precision.
Algorithms like Naive Bayes are direct applications of this theorem, providing a concrete method for building probabilistic classifiers.

So far, we have focused on the probabilistic architecture of learning. In our next lesson, we will shift gears to the mechanics of fitting these models to data. This will lead us into the world of optimization, where we will learn how to find the model parameters that best explain our data by minimizing an error function. Our first tool for this task will be the gradient, and we'll start by reviewing how to compute gradients for multivariable functions.