Gradient Descent for Logistic Regression

The Intuition: Descending the Cost Surface

In the previous lessons, we established a cost function, $J(\mathbf{w},b)$, which measures how "wrong" our model is for a given set of parameters $\mathbf{w}$ and $b$. Our goal is to find the specific values of $\mathbf{w}$ and $b$ that make this cost as low as possible.

Imagine the cost function as a landscape, with hills and valleys. The parameters $\mathbf{w}$ and $b$ are our coordinates in this landscape, and the cost $J$ is the altitude. Our objective is to find the lowest point in this landscape—the global minimum.

Gradient descent is an iterative algorithm that does exactly this. It starts at some random point (randomly initialized parameters) and takes a series of steps "downhill" until it reaches a minimum. But how does it know which way is downhill?

The answer lies in the gradient, $\nabla J$. In calculus, the gradient of a function at a particular point is a vector that points in the direction of the steepest ascent. Therefore, to go downhill most quickly, we should move in the direction opposite to the gradient.

This process is visualized below. At each point on the curve, we calculate the slope (the derivative, which is the 1D version of the gradient). If the slope is negative, we need to increase w. If it's positive, we need to decrease w.

This leads to the core update rule of gradient descent:

Where:

• $\theta$ represents our parameters ($\mathbf{w}$ and $b$).
• $\alpha$ is the learning rate, a hyperparameter that controls the size of each step.
• ${\nabla }_{\theta }J(\theta )$ is the gradient of the cost function with respect to the parameters.

The following reading provides a fantastic, intuitive explanation of this process.

Deriving the Gradient for Logistic Regression

Now for the central task of this lesson. We have the "what" (the update rule) and the "why" (to minimize cost), but we need the "how": what exactly is ${\nabla }_{\theta }J(\theta )$ for our logistic regression model? To find this, we must compute the partial derivatives of the cost function with respect to each parameter, $w_j$ (for each feature $j$) and $b$.

Let's recall the key equations from our last lesson for a single training example $(\mathbf{x},y)$:

1. Linear Model Output: $z=\mathbf{w}\cdot \mathbf{x}+b$
2. Predicted Probability (Hypothesis): $\hat{y}=\sigma (z)=\frac{1}{1+e^{-z}}$
3. Loss Function (Binary Cross-Entropy): $L(\hat{y},y)=-[y\log (\hat{y})+(1-y)\log (1-\hat{y})]$

Our goal is to find $\frac{\partial L}{\partial w_j}$ and $\frac{\partial L}{\partial b}$. Since $L$ is a function of $\hat{y}$, which is a function of $z$, which is a function of $w_j$ and $b$, we must use the chain rule.

This may look complex, but we can solve it piece by piece. The final result is remarkably simple and elegant. Given your strong math background, you are well-equipped to follow the formal derivation. The following section from the Jurafsky & Martin textbook provides a rigorous and complete walkthrough.

After following that derivation, you will have seen that for a single training example:

And similarly for the bias (where the corresponding "feature" $x_0$ is implicitly 1):

This result is beautifully intuitive. The gradient for a weight $w_j$ is simply the prediction error $(\hat{y}-y)$ scaled by the corresponding input feature value $x_j$. If the prediction is perfect ($\hat{y}=y$), the error is zero, and the gradient is zero, so the weights are not updated. If the error is large, the update will be proportionally large.

For the entire training set of $m$ examples, the cost function $J$ is the average of the individual losses. Due to the properties of differentiation, the gradient of the average is the average of the gradients:

These are precisely the formulas we used without proof in the previous lesson! In vectorized form, this becomes:

For a more dynamic walkthrough of this same derivation, the following video is an excellent resource.

Implementing the Algorithm

Now that we have derived the gradient, we can implement the full gradient descent algorithm. The process is a direct translation of the mathematics into code.

The training loop proceeds as follows:

1. Initialize parameters $\mathbf{w}$ and $b$ (e.g., to zeros).
2. Repeat for a specified number of iterations:
   a. Forward Pass: Compute the predictions $\hat{\mathbf{y}}$ for all training examples using the current $\mathbf{w}$ and $b$.
   b. Compute Gradients: Calculate $\frac{\partial J}{\partial \mathbf{w}}$ and $\frac{\partial J}{\partial b}$ using the formulas we just derived.
   c. Update Parameters: Apply the update rule:
   • $\mathbf{w}:=\mathbf{w}-\alpha \frac{\partial J}{\partial \mathbf{w}}$
   • $b:=b-\alpha \frac{\partial J}{\partial b}$
3. Return the final, optimized parameters $\mathbf{w}$ and $b$.

Let's integrate this logic into the fit method of the MyLogisticRegression class from our last lesson.

import numpy as np

class MyLogisticRegression:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.learning_rate = learning_rate
        self.n_iterations = n_iterations
        self.weights = None
        self.bias = None

    def _sigmoid(self, z):



        # Clip to avoid overflow/underflow in np.exp
        z = np.clip(z, -500, 500)
        return 1 / (1 + np.exp(-z))

    def fit(self, X, y):



        # 1. Initialize parameters
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0




        # 2. Gradient Descent loop
        for i in range(self.n_iterations):



            # a. Forward Pass: Compute predictions
            linear_model = np.dot(X, self.weights) + self.bias
            y_predicted = self._sigmoid(linear_model)




            # b. Compute Gradients
            # This is the direct implementation of our derived formulas
            dw = (1 / n_samples) * np.dot(X.T, (y_predicted - y))
            db = (1 / n_samples) * np.sum(y_predicted - y)




            # c. Update Parameters
            self.weights -= self.learning_rate * dw
            self.bias -= self.learning_rate * db
            



            # Optional: Print cost to monitor progress
            if i % 100 == 0:
                cost = -np.mean(y * np.log(y_predicted) + (1-y) * np.log(1 - y_predicted))
                print(f"Iteration {i}: Cost {cost:.4f}")

    def predict_proba(self, X):
        linear_model = np.dot(X, self.weights) + self.bias
        return self._sigmoid(linear_model)

    def predict(self, X, threshold=0.5):
        probabilities = self.predict_proba(X)
        return [1 if i > threshold else 0 for i in probabilities]

Take a moment to study the fit method. You can now see every line has a clear mathematical justification based on our derivation. The code is no longer a black box; it's a direct implementation of the optimization principle we've just explored.

The following video shows a similar from-scratch Python implementation, which can help reinforce your understanding of how the mathematical formulas translate into NumPy operations.

A Note on Gradient Descent Variants

The version of gradient descent we implemented is called Batch Gradient Descent, because it calculates the gradient using the entire batch of training data in each iteration. This provides a very accurate estimate of the gradient, leading to a smooth convergence. However, it can be computationally expensive for very large datasets.

Two other common variants are:

• Stochastic Gradient Descent (SGD): Calculates the gradient and updates the parameters using only one randomly chosen training example at a time. This is much faster per update, but the path to the minimum is much noisier.
• Mini-Batch Gradient Descent: A compromise between the two. It computes the gradient on a small, random subset (a "mini-batch") of the data. This is the most common approach used in modern machine learning and deep learning, as it balances computational efficiency with stable convergence.

You can read more about these variants in the Jurafsky & Martin text, which provides a concise comparison.

Conclusion

In this lesson, you have moved from being a user of an optimization algorithm to understanding its fundamental mechanics. We have demystified the training process of logistic regression by deriving its gradient from first principles.

Here are the key takeaways:

• Gradient Descent is an iterative optimization algorithm that minimizes a cost function by repeatedly taking steps in the direction of the negative gradient.
• The learning rate, $\alpha$, is a crucial hyperparameter that controls the step size.
• Using the chain rule, we derived the gradient of the Binary Cross-Entropy loss for logistic regression, which results in the elegant formula: $\nabla J\propto (\text{error})\cdot (\text{input})$.
• The derived formulas can be implemented in a straightforward training loop to find the optimal parameters for our model.

With this foundation, you now have a complete understanding of one of the most fundamental classification algorithms in machine learning. In our next lesson, we will address a common problem in model training: overfitting. We will explore regularization, a technique used to prevent models from becoming too complex and to improve their ability to generalize to new, unseen data.