Multi-Class Logistic Regression with Softmax

Hello! Welcome to the next lesson in your journey to master statistics for machine learning.

In our last session, we delved into the interpretation of logistic regression coefficients, learning how to explain a model's predictions using odds ratios. We saw how the sigmoid function takes a linear score and elegantly maps it to a probability for a binary (two-class) outcome.

Today, we'll build directly on that foundation to tackle a much broader set of problems. What if your classification task isn't a simple "yes" or "no"? What if you need to classify a news article as 'Sports', 'Politics', or 'Technology', or identify a handwritten digit from 0 through 9? This is where multi-class classification comes in.

This lesson's goal is to equip you to extend logistic regression to multi-class problems using the softmax function. The softmax function is a beautiful generalization of the sigmoid function and is a fundamental building block in modern AI, especially in the neural networks you're interested in.

From Binary to Multi-Class: A First Attempt

Let's say we have a problem with three classes: 'No Infection', 'Viral Infection', and 'Bacterial Infection'. A natural first thought might be to just use what we already know: binary logistic regression. We could build three separate models in a strategy called One-vs-All (or One-vs-Rest):

Model 1: 'No Infection' (1) vs. 'Viral' or 'Bacterial' (0).
Model 2: 'Viral Infection' (1) vs. 'No' or 'Bacterial' (0).
Model 3: 'Bacterial Infection' (1) vs. 'No' or 'Viral' (0).

When a new patient arrives, we'd run their data through all three models and get three separate probabilities. But there's a problem with this approach, which the following video explains well.

Multinomial logistic regression | softmax regression | explained

This video from TileStats demonstrates the One-vs-All approach and highlights its key limitation. It then introduces the softmax function as the solution.

Please watch from 04:00 to 07:48. Focus on: How three separate binary models are created. The problem that arises when you look at the output probabilities from the three models (they don't sum to 1). How the softmax function is introduced to fix this by converting the model outputs (log-odds) into a valid probability distribution.

As the video showed, the One-vs-All approach gives us three scores, but they aren't true probabilities that collectively describe the chances for all classes. The sum of the "probabilities" from the three models was greater than 1, which doesn't make sense. We need a way to take the raw output scores from our models and transform them into a set of probabilities that sum to 1. This is precisely the job of the softmax function.

The Softmax Function: A Deeper Look

The softmax function is a generalization of the sigmoid function. While sigmoid is perfect for two classes, softmax handles two or more. It takes a vector of K real-numbered scores (often called logits) and transforms them into a vector of K probabilities.

For K classes, we first calculate a separate score, $z_{i}$ , for each class using a linear model:
$z_{i} = w_{i} \cdot x + b_{i}$
This gives us a vector of scores $z = [z_{1}, z_{2}, \dots, z_{K}]$ .

The softmax function then calculates the probability for class i as:
$P (y = i ∣ x) = softmax (z_{i}) = \frac{e ^{z_{i}}}{\sum _{j = 1}^{K} e ^{z_{j}}}$

Let's break down what this formula does:

Exponentiate: It takes the exponential of each score ( $e^{z_{i}}$ ). This forces all values to be positive. A larger score $z_{i}$ will result in a much larger positive value.
Normalize: It divides each exponentiated score by the sum of all exponentiated scores. This step ensures that the resulting probabilities add up to 1.

The following video provides an excellent, clear walkthrough of this calculation with a concrete numerical example.

Softmax Regression (C2W3L08)

In this classic video from DeepLearningAI, Andrew Ng explains the mechanics of the softmax layer in a neural network, which is identical to its use in softmax regression. This will give you a solid, practical understanding of the calculation.

Watch from 02:55 to 08:40. Pay close attention to: How the linear scores (the Z vector) are calculated. The step-by-step process of exponentiating and then normalizing to get the final probabilities (the A or Y-hat vector). The key insight that the softmax activation function takes a vector as input and produces a vector as output.

The Softmax Regression Model

Now we can assemble the full model, which is also known as Multinomial Logistic Regression.

Logistic Regression vs. Softmax Regression Architectures — This diagram compares the architecture of binary logistic regression (left) with softmax regression (right). While logistic regression uses a single set of weights and a sigmoid function for a binary output, softmax regression uses K sets of weights to produce K scores (logits), which are then passed through the softmax function to generate K probabilities.

The process is as follows:

For a given input $x$ , calculate K scores (logits), one for each class. This requires K separate weight vectors and biases.
Feed this vector of K scores into the softmax function.
The output is a vector of K probabilities, which sum to 1.
To make a final prediction, we simply choose the class with the highest probability (this operation is called argmax).

Training the Model: Cross-Entropy Loss

To find the optimal weights and biases, we need a loss function that measures how "wrong" our model's predictions are. For multi-class classification, we use Cross-Entropy Loss. It is the direct generalization of the Log Loss we used for binary logistic regression.

Deep Dive into Softmax Regression

This article, 'Deep Dive into Softmax Regression', provides a clear explanation of cross-entropy loss and its relationship to the model's likelihood.

Read the section titled 'Cross-Entropy Loss'. Focus on understanding: The concept of one-hot encoding for the true labels. The formula for cross-entropy loss and how to interpret it. The key takeaway: minimizing cross-entropy loss is equivalent to maximizing the log-likelihood of the model, which means finding the parameters that make the observed data most probable.

Test your understanding!

Suppose a model for classifying an image as a 'cat', 'dog', or 'bird' outputs the following logit scores: $z = [2.0, 1.0, 0.1]$ .

Calculate the probability for each class using the softmax function. You can use a calculator.

(Note: $e^{2.0} \approx 7.39$ , $e^{1.0} \approx 2.72$ , $e^{0.1} \approx 1.11$ )

Show answer

Exponentiate the logits:
- $e^{z_{c a t}} = e^{2.0} \approx 7.39$
- $e^{z_{d o g}} = e^{1.0} \approx 2.72$
- $e^{z_{bi r d}} = e^{0.1} \approx 1.11$
Calculate the sum (the normalizer):
- Sum = $7.39 + 2.72 + 1.11 = 11.22$
Divide each exponentiated logit by the sum:
- $P (cat) = \frac{7.39}{11.22} \approx 0.66$
- $P (dog) = \frac{2.72}{11.22} \approx 0.24$
- $P (bird) = \frac{1.11}{11.22} \approx 0.10$

The resulting probabilities are approximately [0.66, 0.24, 0.10]. They sum to 1.0, and the model would predict 'cat' as it has the highest probability.

Softmax Regression in Practice

One of the great things about modern machine learning libraries is that they make implementing these models straightforward. You don't need to code the softmax function or cross-entropy loss from scratch. scikit-learn's LogisticRegression class is smart enough to handle this automatically.

Deep Dive into Softmax Regression

Let's return to the 'Deep Dive into Softmax Regression' article to see how this works in practice with a real-world dataset.

Read the sections 'Softmax Regression in Scikit-Learn' and 'Example: Classifying Handwritten Digits'. You don't need to run the code yourself, but focus on: How the LogisticRegression object automatically switches to softmax mode for multi-class problems. The process of loading data, training the model (.fit()), and evaluating it (.score()). The example uses the famous MNIST dataset of handwritten digits, a classic multi-class problem.

This practical example shows that the concepts we've discussed are not just theoretical; they are directly implemented in the tools you will use to build powerful classifiers.

Conclusion

Congratulations! You've successfully made the leap from binary to multi-class classification. This is a significant step that dramatically expands the range of problems you can solve.

Key Takeaways:

Softmax Regression (or Multinomial Logistic Regression) generalizes logistic regression to problems with more than two classes.
It computes a separate linear score (logit) for each class.
The softmax function is the crucial component that converts this vector of scores into a valid probability distribution, where all outputs are between 0 and 1 and sum to 1.
The model is trained by finding the weights that minimize the cross-entropy loss, which measures the difference between the predicted probabilities and the true (one-hot encoded) labels.
In libraries like scikit-learn, using softmax regression is often as simple as applying the standard LogisticRegression classifier to a dataset with a multi-class target variable.

Preview of the Next Lesson:
We've now built models for both binary and multi-class classification. But how do we know if these models are any good? A simple accuracy score can be misleading, especially if some classes are much rarer than others. In the next lesson, you will learn how to evaluate a classifier's performance using a confusion matrix, precision, recall, and F1-score. These tools will give you a much deeper and more reliable understanding of your model's strengths and weaknesses.