Implementing One-vs-All for Multi-Class Classification

1. The One-vs-All (OvA) Concept

The core idea of One-vs-All (also called One-vs-Rest or OvR) is simple: we decompose a single multi-class problem into multiple binary classification problems.

If you have K different classes, you will train K separate binary classifiers. Each classifier is trained to distinguish one specific class from all the others combined.

• Classifier 1: Learns to distinguish "Class 1" (positive class) from "Not Class 1" (negative class).
• Classifier 2: Learns to distinguish "Class 2" (positive class) from "Not Class 2".
• ...and so on, up to Classifier K.

When you want to classify a new, unseen data point, you pass it to all K classifiers. Each one will output a probability score indicating its confidence that the data point belongs to its "special" class. The final prediction is simply the class whose classifier returned the highest probability.

The following videos provide an excellent conceptual overview of this strategy.

After training, you'll effectively have K different decision boundaries partitioning your feature space.

Each classifier learns a single boundary to separate its target class from the rest. For a new point, the classifier that is 'most confident' (highest probability) determines the final label.

2. Implementation Strategy

Given your preference for building from scratch, we will create a OneVsAll wrapper class that uses your existing LogisticRegression class as its building block. Here's our plan:

1. Data Transformation: The most critical step is preparing the target labels. If your original y vector has labels [0, 1, 2, 0, 1, 2, ...], you will need to create K new binary target vectors. For a 3-class problem, it would look like this:
   • y_for_class_0: [1, 0, 0, 1, 0, 0, ...]
   • y_for_class_1: [0, 1, 0, 0, 1, 0, ...]
   • y_for_class_2: [0, 0, 1, 0, 0, 1, ...]

*This image from a tutorial perfectly illustrates the data transformation step. A single 'Target' column is converted into multiple binary columns, one for each unique class, which is exactly what we need to do.*

2. Training Loop: We'll create a fit method that iterates from c = 0 to K-1. In each iteration, it will:
* Generate the appropriate binary y vector for class c.
* Instantiate and train a new LogisticRegression model on that binary data.
* Store the trained model (specifically, its weights and bias).

3. Prediction: The predict method will take new data X and:
   • Calculate the probability scores for each sample from all K stored classifiers. This will result in a matrix of shape (n_samples, K).
   • Find the index of the maximum probability for each sample using np.argmax(). This index corresponds to the predicted class label.

This approach is highly modular. You could swap out LogisticRegression with another binary classifier (like the SVM you'll build later) and the OneVsAll wrapper would still work!

For a detailed walkthrough of this implementation logic, the following blog post is an excellent guide. It uses a very clean, vectorized approach that leverages NumPy well.

3. Implementing One-vs-All From Scratch

Let's put this plan into action. First, we need to slightly modify your LogisticRegression class from the previous lesson to provide probability scores, not just the final class labels. This is crucial for the OvA strategy.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler

# --- Step 1: Update your LogisticRegression class ---
# We add a `predict_proba` method.
class LogisticRegression:
    def __init__(self, learning_rate=0.01, iterations=1000):
        self.lr = learning_rate
        self.iterations = iterations
        self.weights = None
        self.bias = 0

    def _sigmoid(self, z):
        # Clip to avoid overflow/underflow in np.exp
        z = np.clip(z, -500, 500)
        return 1 / (1 + np.exp(-z))

    def fit(self, X, y):
        m, n = X.shape
        self.weights = np.zeros(n)
        self.bias = 0
        for _ in range(self.iterations):
            z = np.dot(X, self.weights) + self.bias
            h = self._sigmoid(z)
            dw = (1/m) * np.dot(X.T, (h - y))
            db = (1/m) * np.sum(h - y)
            self.weights -= self.lr * dw
            self.bias -= self.lr * db

    def predict_proba(self, X):
        """Returns the probability of belonging to the positive class (class 1)."""
        z = np.dot(X, self.weights) + self.bias
        return self._sigmoid(z)

    def predict(self, X):
        """Returns the predicted class label (0 or 1)."""
        y_pred_proba = self.predict_proba(X)
        y_pred_class = (y_pred_proba >= 0.5).astype(int)
        return y_pred_class

# --- Step 2: Create the OneVsAll wrapper class ---
class OneVsAll:
    def __init__(self, base_classifier, learning_rate=0.01, iterations=1000):
        """
        Initializes the One-vs-All wrapper.
        :param base_classifier: The binary classifier class to be used (e.g., LogisticRegression).
        :param learning_rate: The learning rate for the base classifiers.
        :param iterations: The number of training iterations for the base classifiers.
        """
        self.base_classifier = base_classifier
        self.lr = learning_rate
        self.iterations = iterations
        self.classifiers = []
        self.classes = []

    def fit(self, X, y):
        """
        Trains K binary classifiers, one for each class.
        """
        self.classes = np.unique(y)
        self.classifiers = []
        
        for c in self.classes:
            # Create a binary target vector for the current class `c`
            y_binary = (y == c).astype(int)
            
            # Instantiate and train a new binary classifier
            classifier = self.base_classifier(self.lr, self.iterations)
            classifier.fit(X, y_binary)
            
            # Store the trained classifier
            self.classifiers.append(classifier)

    def predict(self, X):
        """
        Makes predictions by choosing the class with the highest probability.
        """
        # Get probabilities from all classifiers
        # This will be a matrix of shape (n_samples, n_classes)
        all_probas = np.array([clf.predict_proba(X) for clf in self.classifiers]).T
        
        # Find the index of the highest probability for each sample
        highest_proba_indices = np.argmax(all_probas, axis=1)
        
        # Map the indices back to the original class labels
        predictions = self.classes[highest_proba_indices]
        return predictions

# --- Step 3: Test the implementation ---
# Generate a 3-class dataset
X, y = make_classification(n_samples=500, n_features=2, n_redundant=0, 
                           n_informative=2, random_state=42, n_clusters_per_class=1, n_classes=3)

# Scale features for better convergence
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Instantiate and train your OneVsAll classifier
ova_model = OneVsAll(base_classifier=LogisticRegression, learning_rate=0.1, iterations=1000)
ova_model.fit(X_scaled, y)

# Make predictions
y_pred = ova_model.predict(X_scaled)

# Calculate and print accuracy
accuracy = np.mean(y_pred == y)
print(f"Model Accuracy: {accuracy * 100:.2f}%")

# Optional: Visualize the results
plt.figure(figsize=(8, 6))
plt.scatter(X_scaled[:, 0], X_scaled[:, 1], c=y, cmap=plt.cm.RdYlBu, edgecolors='k', label='True Labels')
plt.scatter(X_scaled[y_pred != y, 0], X_scaled[y_pred != y, 1], 
            facecolors='none', edgecolors='red', s=100, label='Misclassified')
plt.title('One-vs-All Classification Results')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.legend()
plt.show()

When you run this code, you should see a high accuracy score, demonstrating that your OneVsAll wrapper successfully trained three separate logistic regression models and is correctly combining their outputs to classify the data.

Conclusion

Congratulations! You have successfully extended your binary classifier to handle multi-class problems. This is a significant step forward, greatly expanding the range of problems you can solve.

Key Takeaways:

• The One-vs-All (OvA) strategy transforms a K-class problem into K binary classification problems.
• The implementation involves creating a wrapper class that trains and manages multiple instances of a base binary classifier.
• The key steps are data transformation (creating binary target vectors) and prediction aggregation (choosing the class with the highest probability via argmax).
• OvA is a general-purpose, modular strategy that can be used with any binary classifier, not just logistic regression.

Preview of the Next Lesson:
While OvA is effective and widely used, it feels a bit like a "patch" on top of a binary model. Each classifier is trained in isolation, unaware of the others. In the next lesson, we will explore a more direct and mathematically elegant approach for multi-class classification: Softmax Regression. You will learn to implement the softmax function, which is a generalization of the sigmoid function, allowing a single model to output a probability distribution across all K classes simultaneously.