Naive Bayes Classification with Mixed Features

Greetings, Detective.

In our last briefing, we brought a powerful new tool to the "Mystery Client" case: the K-Nearest Neighbors algorithm. By mapping our enigmatic applicant in a space of past clients, we judged them by the company they keep. The verdict, based on a majority vote of their closest peers, was a "Poor Fit." This was a classification based on proximity, a democratic but somewhat blunt instrument.

Today, we revisit the evidence with a more nuanced, probabilistic lens. What if not all clues are created equal? A client's score on a creativity assessment and their previous experience in non-traditional relationships are different kinds of evidence. A simple distance metric doesn't capture the unique story each clue tells. Our goal is to re-classify the client using a Naive Bayes model, handling both their discrete and continuous profile features appropriately. We will move from the court of public opinion to the calculated reasoning of a probability theorist, weighing each piece of evidence to determine the most likely truth.

The Logic of Probabilistic Inference

At the core of this new approach is a principle you are well-acquainted with from your studies in Bayesian decision theory: Bayes' theorem. We are not just counting votes; we are updating our belief about the client's potential for success based on the evidence we observe.

The classification question, "What is the probability this client is a 'Successful Engagement', given their profile?" can be expressed as:

P (Class ∣ Evidence) = \frac{P ( Evidence ∣ Class ) \times P ( Class )}{P ( Evidence )}

Since the denominator, $P (Evidence)$ , is the same for all classes, we can simplify our task to finding the class that maximizes the numerator:

Predicted Class = Class argmax [P (Evidence ∣ Class) \times P (Class)]

Where:

$P (Class)$ is the Prior: The overall probability of a client being "Successful" or a "Poor Fit," based on our past records.
$P (Evidence ∣ Class)$ is the Likelihood: The probability of observing this specific client's profile if they truly belonged to that class.
$P (Class ∣ Evidence)$ is the Posterior: The updated probability of them belonging to a class, after we've seen the evidence.

The following video provides an excellent visual and conceptual walkthrough of applying this theorem to a classification problem.

The Math Behind Bayesian Classifiers Clearly Explained!

This video from the channel Normalized Nerd clearly explains how Bayes' theorem is adapted for classification.

Watch the first part of the video, up to where the host introduces the example dataset. Pay close attention to the definition of posterior, prior, likelihood, and evidence, and the key insight that we can ignore the denominator when comparing classes.

The "Naive" Assumption: A Detective's Necessary Shortcut

The main challenge lies in calculating the likelihood, $P (Evidence ∣ Class)$ . If our client profile has many features, finding the probability of that exact combination of features is computationally difficult and often impossible with limited data. Imagine trying to find past clients who have exactly the same creativity score, technical skill, age, and relationship history. You might find none, leading to a likelihood of zero.

To overcome this, we make a "naive" but powerful assumption: all features are conditionally independent of each other, given the class. This means we assume that a client's creativity score tells us nothing about their relationship history, once we already know whether they were a "Successful Engagement" or not.

This assumption allows us to break down the complex likelihood calculation into a simple product of individual likelihoods:

P (X_{1}, X_{2}, ..., X_{n} ∣ Class) = P (X_{1} ∣ Class) \times P (X_{2} ∣ Class) \times \dots \times P (X_{n} ∣ Class)

This simplifies our investigation immensely. We no longer need to find perfect matches, only to assess each piece of evidence on its own merits for a given class.

This diagram illustrates the two primary ways we'll apply this principle. On the left, **Gaussian Naive Bayes** is used for continuous features (like a test score), assuming they follow a normal distribution within each class. On the right, **Multinomial Naive Bayes** is used for discrete features (like word counts or categorical choices), where we count frequencies.

Case File: Re-examining the Mystery Client

Let's apply this to a new, simplified profile for our mystery client. We have two key data points:

'Design Aptitude' Score: A continuous value from 1 to 10.
'Open to Experimentation': A discrete, binary answer (Yes/No).

Mystery Client's Profile:

Aptitude Score: 7.5
Open to Experimentation: No

Historical Client Data (Our Training Set):

Client	Aptitude Score (X₁)	Open to Experimentation (X₂)	Outcome (Y)
1	8.1	Yes	Successful
2	6.5	Yes	Successful
3	9.0	No	Successful
4	7.2	Yes	Successful
5	4.5	No	Poor Fit
6	5.5	Yes	Poor Fit
7	6.0	No	Poor Fit

Our task is to calculate $P (Successful ∣ Evidence)$ and $P (Poor Fit ∣ Evidence)$ and see which is higher.

First, let's calculate the Priors by counting:

Total clients = 7
Successful = 4, Poor Fit = 3
$P (Successful) = 4/7 \approx 0.57$
$P (Poor Fit) = 3/7 \approx 0.43$

Now, we compute the likelihoods for each feature.

1. Continuous Evidence: Gaussian Naive Bayes

For the 'Aptitude Score' (a continuous feature), we assume the scores for each class are normally distributed.

Step 1: Calculate Mean ( $μ$ ) and Variance ( $σ^{2}$ ) for each class.

For 'Successful' class:
- Scores: {8.1, 6.5, 9.0, 7.2}
- Mean ( $μ_{S}$ ): (8.1 + 6.5 + 9.0 + 7.2) / 4 = 7.7
- Variance ( $σ_{S}^{2}$ ): $\frac{( 8.1 - 7.7 ) ^{2} + ( 6.5 - 7.7 ) ^{2} + ( 9.0 - 7.7 ) ^{2} + ( 7.2 - 7.7 ) ^{2}}{4}$ = 0.875
For 'Poor Fit' class:
- Scores: {4.5, 5.5, 6.0}
- Mean ( $μ_{P F}$ ): (4.5 + 5.5 + 6.0) / 3 = 5.33
- Variance ( $σ_{P F}^{2}$ ): $\frac{( 4.5 - 5.33 ) ^{2} + ( 5.5 - 5.33 ) ^{2} + ( 6.0 - 5.33 ) ^{2}}{3}$ = 0.389

Step 2: Calculate Likelihood using the Gaussian PDF.
The formula is: $P (x ∣ y) = \frac{1}{2 π σ _{y}^{2}} exp (- \frac{( x - μ _{y} ) ^{2}}{2 σ _{y}^{2}})$

This looks complex, but it's just plugging in the values we found. The following reading provides a worked example you can follow.

Naive Bayes Classifier example by hand and how to do in ...

This article provides clear, step-by-step manual calculations for both continuous and discrete features. We will use it as a guide.

Find the section "An Example by hand". Focus on the calculation for the 'Income' variable. See how the author first calculates the mean and variance for each class, then plugs a new value (Income = 120) into the Gaussian probability density formula to find the likelihoods P(\text{Income}=120 | \text{Evade=No}) and P(\text{Income}=120 | \text{Evade=Yes}). We will perform the exact same steps for our 'Aptitude Score'.

Following that example, we calculate the likelihood for our client's score of 7.5:

$P (Score = 7.5∣ Successful) = \frac{1}{2 π ( 0.875 )} exp (- \frac{( 7.5 - 7.7 ) ^{2}}{2 ( 0.875 )}) \approx 0.426 \times 0.977 = 0.416$
$P (Score = 7.5∣ Poor Fit) = \frac{1}{2 π ( 0.389 )} exp (- \frac{( 7.5 - 5.33 ) ^{2}}{2 ( 0.389 )}) \approx 0.639 \times 0.002 = 0.001$

The aptitude score of 7.5 is much more likely to be seen in a "Successful" client than a "Poor Fit".

2. Discrete Evidence and the Zero-Frequency Problem

For the 'Open to Experimentation' feature, we just count. Our client's value is 'No'.

Among 'Successful' clients, 1 out of 4 said 'No'. So, $P (Exp=No ∣ Successful) = 1/4 = 0.25$ .
Among 'Poor Fit' clients, 2 out of 3 said 'No'. So, $P (Exp=No ∣ Poor Fit) = 2/3 \approx 0.67$ .

But what if, in our training data, zero "Successful" clients had said 'No'? The probability would be 0, and this would zero out our entire posterior calculation for that class, regardless of other evidence. This is the zero-frequency problem.

To prevent this, we use a technique called Laplace Smoothing (or add-one smoothing). We add a small value (typically 1, hence 'add-one') to every count.

This image illustrates the process. After computing class priors (Step 1), we compute feature counts for each class (Step 2). In Step 3, we add a smoothing factor $\alpha$ (alpha) to the numerator and adjust the denominator accordingly to prevent zero probabilities.

Let's re-calculate with Laplace smoothing ( $α = 1$ ):
The formula becomes: $P (feature ∣ class) = \frac{count ( feature \cap class ) + α}{count ( class ) + α \times k}$ , where k is the number of possible values for the feature (in this case, 2: Yes/No).

$P (Exp=No ∣ Successful) = \frac{1 + 1}{4 + ( 1 \times 2 )} = \frac{2}{6} \approx 0.33$
$P (Exp=No ∣ Poor Fit) = \frac{2 + 1}{3 + ( 1 \times 2 )} = \frac{3}{5} = 0.60$

Smoothing gives us more robust estimates, especially with small datasets.

The Final Verdict

Now we combine our evidence using the naive independence assumption.

For 'Successful' class:
- Posterior $\propto P (Score=7.5 ∣ S) \times P (Exp=No ∣ S) \times P (S)$
- Posterior $\propto 0.416 \times 0.33 \times 0.57 = 0.078$
For 'Poor Fit' class:
- Posterior $\propto P (Score=7.5 ∣ PF) \times P (Exp=No ∣ PF) \times P (PF)$
- Posterior $\propto 0.001 \times 0.60 \times 0.43 = 0.00026$

Comparing the two results, 0.078 is significantly larger than 0.00026. The evidence overwhelmingly points to one conclusion.

Conclusion: The Naive Bayes analysis classifies the mystery client as a likely "Successful Engagement". This contradicts our KNN result, highlighting how different analytical frameworks can interpret the same data in profoundly different ways.

Naive Bayes Classifier - How to Successfully Use It in Python?

In a real investigation with more complex data, we wouldn't do this by hand. This article demonstrates two powerful, practical approaches for handling mixed data types (continuous and categorical) in Python.

Focus on the last two examples in the article: Approach 1, which involves converting continuous variables into categorical bins, and Approach 2, which involves building separate models for each data type and combining their predictions. This second method is a form of model stacking, a sophisticated technique you'll appreciate.

Conclusion

We have now added a second, more statistically rigorous classification method to our detective's toolkit. By leveraging Bayes' theorem and the practical shortcut of the naive independence assumption, we can weigh different kinds of evidence probabilistically. This method not only provides a classification but also a degree of certainty based on the likelihood of the evidence under each hypothesis.

Key Takeaways:

Naive Bayes is a probabilistic classifier based on applying Bayes' theorem with a "naive" assumption of conditional independence between features.
Continuous features are typically handled using Gaussian Naive Bayes, which assumes the feature values for each class are normally distributed.
Discrete features are handled using Multinomial or Bernoulli Naive Bayes, which relies on frequency counts.
Laplace (add-alpha) smoothing is crucial for discrete features to prevent the zero-frequency problem, where an unobserved feature combination would incorrectly result in a zero posterior probability.
The final classification is determined by the class with the highest maximum a posteriori (MAP) probability.

Preview of the Next Lesson

Our investigation into the mystery client has yielded conflicting verdicts: KNN suggests a "Poor Fit," while Naive Bayes points to a "Successful Engagement." KNN is intuitive but sensitive to the local neighborhood, while Naive Bayes is probabilistic but relies on a strong independence assumption.

What if we wanted a model that was not only predictive but also highly interpretable? A model that could give us a set of simple, human-readable rules to explain its decision? In our next session, we will construct a Decision Tree, creating a flowchart from the data to classify our client and, perhaps, finally resolve this case.

Recommended resources