Soft-Margin SVM with Gradient Descent

1. Assembling the SVM Class

Just like with other models, we'll structure our SVM within a Python class. This keeps our parameters, hyperparameters, and methods organized. Our class will look like this:

import numpy as np

class SVM:
    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iters=1000):
        # Hyperparameters
        self.lr = learning_rate
        self.lambda_param = lambda_param
        self.n_iters = n_iters
        
        # Parameters
        self.w = None  # Weight vector
        self.b = None  # Bias term

    def fit(self, X, y):
        # Gradient descent logic goes here
        pass

    def predict(self, X):
        # Prediction logic goes here
        pass

The __init__ method stores the hyperparameters:

• learning_rate: Controls the step size in gradient descent.
• lambda_param: The regularization parameter. It balances the trade-off between maximizing the margin (a smaller $\lambda$) and minimizing classification error (a larger $\lambda$).
• n_iters: The number of training epochs.

2. Implementing the fit Method with Gradient Descent

The fit method is where the learning happens. We'll use Stochastic Gradient Descent (SGD) to train our model. The process iterates through our training data and, for each sample, updates the model's parameters (w and b) using the gradients we derived in the last lesson.

The following video provides an excellent walkthrough of building an SVM class from scratch. As you watch, focus on the overall structure and the implementation of the fit and predict methods.

A quick note on notation: This video uses a slightly non-standard w.x - b for the decision function, whereas we've been using the more common w.x + b. The underlying logic is the same, but the sign of the bias b and its update rule will be inverted. We will stick to our standard w.x + b formulation. The video is still invaluable for understanding the code structure.

Now, let's implement the fit method using the standard w.x + b formulation and the gradients we derived in the previous lesson.

Recall the Gradient Rules:
Let $J$ be the objective function for a single sample $({\mathbf{x}}_i,y_i)$.

• If $y_i(\mathbf{w}\cdot {\mathbf{x}}_i+b)\ge 1$ (correctly classified, outside margin):
  ${\nabla }_{\mathbf{w}}J=\lambda \mathbf{w}$
  ${\nabla }_{b}J=0$
• If $y_i(\mathbf{w}\cdot {\mathbf{x}}_i+b)<1$ (violates margin):
  ${\nabla }_{\mathbf{w}}J=\lambda \mathbf{w}-y_i{\mathbf{x}}_i$
  ${\nabla }_{b}J=-y_i$

And the Gradient Descent Update Rules:
$\mathbf{w}\leftarrow \mathbf{w}-\alpha {\nabla }_{\mathbf{w}}J$
$b\leftarrow b-\alpha {\nabla }_{b}J$
where $\alpha$ is the learning rate.

Here is the full implementation of the fit method:

# Inside the SVM class

def fit(self, X, y):
    n_samples, n_features = X.shape
    
    # Ensure labels are -1 or 1 for the hinge loss math to work
    y_ = np.where(y <= 0, -1, 1)
    
    # Initialize parameters
    # Given your background, you know initializing weights to zero is simple but
    # random initialization can sometimes be better. For a convex problem like
    # linear SVM, zero-initialization is fine.
    self.w = np.zeros(n_features)
    self.b = 0
    
    # Gradient Descent loop
    for _ in range(self.n_iters):
        # Loop over each individual data sample (Stochastic Gradient Descent)
        for idx, x_i in enumerate(X):
            y_i = y_[idx]
            
            # Check the margin condition
            condition = y_i * (np.dot(x_i, self.w) + self.b) >= 1
            
            if condition:
                # Gradient of the regularizer only
                dw = self.lambda_param * self.w
                db = 0
            else:
                # Gradient of regularizer + gradient of hinge loss
                dw = self.lambda_param * self.w - y_i * x_i
                db = -y_i
            
            # Update the parameters
            self.w -= self.lr * dw
            self.b -= self.lr * db

3. Implementing the predict Method

Once the model is trained, self.w and self.b define our optimal hyperplane. To make a prediction for a new data point $\mathbf{x}$, we simply calculate its position relative to the hyperplane.

The decision function is $f(\mathbf{x})=\mathbf{w}\cdot \mathbf{x}+b$.

• If $f(\mathbf{x})\ge 0$, we classify it as class +1.
• If $f(\mathbf{x})<0$, we classify it as class -1.

The np.sign() function does exactly this.

# Inside the SVM class

def predict(self, X):
    # Calculate the dot product and add the bias
    linear_output = np.dot(X, self.w) + self.b
    
    # Return the sign of the output. np.sign returns -1 for <0, 1 for >0.
    # We use np.where to map any 0 outputs to -1, which is a common convention.
    return np.where(np.sign(linear_output) == 0, -1, np.sign(linear_output))

4. Putting It All Together and Testing

Now you have all the components. Let's combine them into a single script and test our SVM on a synthetic dataset. This is the moment where theory and code come together to create a working model.

Here is the complete SVM class and a testing script:

import numpy as np
from sklearn import datasets
import matplotlib.pyplot as plt

class SVM:
    def __init__(self, learning_rate=0.001, lambda_param=0.01, n_iters=1000):
        self.lr = learning_rate
        self.lambda_param = lambda_param
        self.n_iters = n_iters
        self.w = None
        self.b = None

    def fit(self, X, y):
        n_samples, n_features = X.shape
        y_ = np.where(y <= 0, -1, 1)
        
        self.w = np.zeros(n_features)
        self.b = 0
        
        for _ in range(self.n_iters):
            for idx, x_i in enumerate(X):
                y_i = y_[idx]
                condition = y_i * (np.dot(x_i, self.w) + self.b) >= 1
                
                if condition:
                    dw = self.lambda_param * self.w
                    db = 0
                else:
                    dw = self.lambda_param * self.w - y_i * x_i
                    db = -y_i
                
                self.w -= self.lr * dw
                self.b -= self.lr * db

    def predict(self, X):
        linear_output = np.dot(X, self.w) + self.b
        return np.where(np.sign(linear_output) == 0, -1, np.sign(linear_output))

# --- Testing our SVM ---
if __name__ == "__main__":
    # Generate synthetic data
    X, y = datasets.make_blobs(
        n_samples=50, n_features=2, centers=2, cluster_std=1.05, random_state=40
    )
    # Convert labels from {0, 1} to {-1, 1}
    y = np.where(y == 0, -1, 1)

    # Instantiate and train the SVM
    clf = SVM()
    clf.fit(X, y)
    
    # Make predictions (optional, for accuracy check)
    predictions = clf.predict(X)
    accuracy = np.mean(predictions == y)
    print(f"SVM classification accuracy: {accuracy * 100:.2f}%")

    # Visualization
    def visualize_svm():
        plt.figure(figsize=(10, 6))
        plt.scatter(X[:, 0], X[:, 1], marker="o", c=y, s=30, cmap=plt.cm.Paired)

        # Plot the decision boundary
        ax = plt.gca()
        xlim = ax.get_xlim()
        ylim = ax.get_ylim()

        xx = np.linspace(xlim[0], xlim[1], 30)
        yy = np.linspace(ylim[0], ylim[1], 30)
        YY, XX = np.meshgrid(yy, xx)
        xy = np.vstack([XX.ravel(), YY.ravel()]).T
        Z = np.dot(xy, clf.w) + clf.b
        Z = Z.reshape(XX.shape)

        # Plot decision boundary and margins
        ax.contour(
            XX, YY, Z, colors="k", levels=[-1, 0, 1], alpha=0.5, linestyles=["--", "-", "--"]
        )
        
        # Highlight support vectors (points on or inside the margin)
        margin_condition = y * (np.dot(X, clf.w) + clf.b) <= 1
        ax.scatter(X[margin_condition, 0], X[margin_condition, 1], s=100,
                   linewidth=1, facecolors='none', edgecolors='k')
                   
        plt.title("Linear SVM with Soft Margin from Scratch")
        plt.xlabel("Feature 1")
        plt.ylabel("Feature 2")
        plt.show()

    visualize_svm()

When you run this script, you will see a plot showing your data points, the decision boundary (solid line) your SVM has learned, and the margins (dashed lines). The points that are circled are the support vectors—the ones that are either on the margin or violate it, and thus actively define the boundary.

Conclusion

Congratulations! You have successfully implemented a linear Support Vector Machine from scratch. You've taken the core concepts of margin maximization and hinge loss, translated them into a gradient-based optimization algorithm, and built a working classifier.

Key Takeaways:

• The fit method implements gradient descent by iterating through data, calculating gradients based on the margin condition, and updating the w and b parameters.
• The predict method uses the trained w and b to calculate the sign of the decision function $\mathbf{w}\cdot \mathbf{x}+b$, which determines the class label.
• The regularization parameter $\lambda$ provides a crucial knob to control the "softness" of the margin, allowing the model to balance between a wide margin and misclassifying some points—a key defense against overfitting.

Preview of the Next Lesson:

Our current SVM can only find linear decision boundaries. What happens when the data is not linearly separable? In the next lesson, we will explore one of the most powerful ideas in machine learning: the kernel trick. You will learn how to implement kernel functions (like the polynomial and RBF kernels) to project data into higher dimensions, allowing your linear SVM to find complex, non-linear decision boundaries without changing the core optimization algorithm you just built.