Welcome back! In our previous lessons, we've explored the theoretical underpinnings of the Naive Bayes algorithm, from its reliance on Bayes' theorem to the specific assumptions made by its three main variants: Gaussian, Multinomial, and Bernoulli. You now have the conceptual framework to understand why and when to use each type.
Today, we transition from theory to practice. Our goal is to apply this knowledge to build, train, and evaluate Naive Bayes classifiers on real-world datasets using Python and scikit-learn. We will tackle two distinct problems: first, a classification task with continuous numerical data, which is a perfect fit for Gaussian Naive Bayes; and second, a text classification task (spam detection), where Multinomial Naive Bayes shines. Along the way, we'll expand our model evaluation toolkit beyond simple accuracy to include more nuanced and powerful metrics.
Applying Gaussian Naive Bayes to Numerical Data
Let's begin with a classic machine learning problem: predicting survival on the Titanic. The dataset's features, such as age and fare, are continuous, making Gaussian Naive Bayes the appropriate choice. Recall from our last lesson that this model assumes features follow a Gaussian (normal) distribution for each class.
The process of applying this model involves several familiar steps: data loading, cleaning, preprocessing, splitting, training, and evaluation. The following video from the codebasics channel provides an excellent walkthrough.
Machine Learning Tutorial Python - 14: Naive Bayes Classifier Algorithm Part 1
This video, "Machine Learning Tutorial Python - 14: Naive Bayes Classifier Algorithm Part 1," will guide us through implementing Gaussian Naive Bayes in Python.
Please watch the following segments: Data Preprocessing (this segment): Observe the standard preprocessing steps, including dropping irrelevant columns, using one-hot encoding for the 'Sex' column, and imputing missing 'Age' values with the mean. This should be a good review of concepts from our initial modules. Model Training (this part): This section shows how to perform the train-test split and then train the GaussianNB model from scikit-learn with a single .fit() command. Prediction and Scoring (this final part): Watch how the trained model is used to make predictions and how its accuracy is calculated with the .score() method. Pay attention to the use of .predict_proba(), which reveals the underlying probabilities the model calculates for each class—the core of its decision-making process.
As you saw, implementing the model is quite straightforward with scikit-learn. Most of the work lies in preparing the data so the model can process it effectively. The model achieved an accuracy of around 78-81%. But is accuracy the whole story?
Evaluating Classifier Performance Beyond Accuracy
Accuracy tells us the overall fraction of correct predictions, but it can be a deceptive metric, especially when dealing with imbalanced classes (e.g., a dataset with 95% non-spam and 5% spam emails). A model that always predicts "non-spam" would be 95% accurate but completely useless. To get a more complete picture, we need a better tool: the confusion matrix.
A confusion matrix is a table that breaks down the performance of a classification model by showing the counts of correct and incorrect predictions for each class.
The four components are:
- True Positives (TP): The model correctly predicted the positive class. (e.g., correctly identified a sick person).
- True Negatives (TN): The model correctly predicted the negative class. (e.g., correctly identified a healthy person).
- False Positives (FP): The model incorrectly predicted the positive class. (Also known as a "Type I error"). (e.g., incorrectly identified a healthy person as sick).
- False Negatives (FN): The model incorrectly predicted the negative class. (Also known as a "Type II error"). (e.g., incorrectly identified a sick person as healthy).
From these four values, we can derive more informative metrics:
-
Precision: What proportion of positive identifications was actually correct?
High precision means the model has a low false positive rate. It is a good metric to use when the cost of a false positive is high (e.g., flagging a legitimate email as spam). -
Recall (Sensitivity): What proportion of actual positives was identified correctly?
High recall means the model has a low false negative rate. It is crucial when the cost of a false negative is high (e.g., failing to detect a fraudulent transaction or a serious disease). -
F1-Score: The harmonic mean of precision and recall. It provides a single score that balances both concerns.
The following Kaggle notebook provides a practical guide to implementing these metrics in scikit-learn.
Naive Bayes Classifier in Python
This notebook demonstrates how to evaluate a Naive Bayes classifier using the full suite of metrics.
Please navigate through the notebook and read the following sections: Section 15: Confusion matrix. Read the explanation and review the code for generating a confusion matrix and visualizing it with a heatmap. Section 16: Classification metrices. Study the code for printing a classification report, which conveniently summarizes precision, recall, and F1-score for each class. Section 18: ROC - AUC. Review the concept of the ROC Curve and AUC. The ROC curve plots the True Positive Rate vs. the False Positive Rate, and the Area Under the Curve (AUC) gives a single number that measures the model's ability to separate the classes. An AUC of 1.0 is perfect, while 0.5 is no better than random guessing.
Applying Multinomial Naive Bayes to Text Data
Now, let's shift to a domain where Naive Bayes is famously effective: text classification. We'll build a spam filter using Multinomial Naive Bayes, which is designed for discrete count data—like the frequency of words in an email.
The primary challenge with text is that machine learning models only understand numbers. We must first convert the text into a numerical format. A common technique for this is the bag-of-words model, implemented in scikit-learn using CountVectorizer. This tool creates a vocabulary of all unique words in the text corpus and represents each document as a vector of word counts.
The second part of the codebasics tutorial expertly demonstrates this entire process, and introduces a powerful concept for streamlining your workflow: scikit-learn Pipelines.
Machine Learning Tutorial Python - 15: Naive Bayes Classifier Algorithm Part 2
This video, "Machine Learning Tutorial Python - 15: Naive Bayes Classifier Algorithm Part 2," shows how to build a spam detector.
Please watch the following segments: Text to Numbers (this part): The video starts by loading the data and converting the categorical labels ('ham'/'spam') to numbers. Crucially, it then explains and implements CountVectorizer to transform the email text into a numerical matrix. Training the Model (this segment): Here, the presenter explains why MultinomialNB is the right choice for this word-count data. He then trains the model and tests it on a couple of sample emails, showing its impressive accuracy. Using a Pipeline (this final segment): This is a key section. The presenter introduces sklearn.pipeline.Pipeline to chain the CountVectorizer and MultinomialNB classifier into a single object. This is a best practice that simplifies your code, prevents data leakage from the test set, and makes it much easier to make predictions on new, raw text.
The use of a Pipeline is a significant step up in building robust machine learning workflows. It encapsulates the entire sequence of data transformation and model fitting, ensuring that the same steps are consistently applied during both training and prediction. Given your background in software development, you can likely appreciate how this modular approach improves code organization and reduces errors.
Conclusion
In this lesson, you have successfully bridged the gap between Naive Bayes theory and practice. You've implemented and evaluated two of its most common variants on appropriate datasets, learning the essential preprocessing steps for both numerical and text data.
Key Takeaways:
scikit-learnoffers simple, high-level APIs likeGaussianNBandMultinomialNBfor applying Naive Bayes classifiers.- For continuous numerical data, Gaussian Naive Bayes is the go-to variant, but requires standard data cleaning and preprocessing.
- For text classification, Multinomial Naive Bayes is a strong baseline. It requires converting text to numerical vectors, typically using
CountVectorizer. - Evaluating a classifier requires a nuanced approach. While accuracy is a starting point, the confusion matrix, precision, recall, F1-score, and ROC-AUC provide a much deeper understanding of a model's performance.
scikit-learnPipelines are an essential tool for creating robust, reusable, and less error-prone machine learning workflows by chaining preprocessing and modeling steps together.
In our next lesson, we will explore a completely different family of algorithms: tree-based models. We'll start with the fundamental Decision Tree, a highly interpretable model that makes predictions by learning a set of simple, hierarchical rules from the data.