Logistic Regression Gradient Derivation

Hello! Welcome back to our series on the mathematical foundations of machine learning.

In our last lesson, we successfully derived the objective function for logistic regression: the average Binary Cross-Entropy Loss, $J(\mathbf{\theta })$. We established that minimizing this function is equivalent to finding the Maximum Likelihood Estimate for our model's parameters. Our cost function is:
$J(\mathbf{\theta })=-\frac{1}{n}{\sum }_{i=1}^n\left[y^{(i)}\log (h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))\right]$
where $h_{\mathbf{\theta }}(\mathbf{x})=\sigma ({\mathbf{\theta }}^T\mathbf{x})$.

Today, we will tackle the next crucial step: deriving the gradient of this objective function, ${\nabla }_{\mathbf{\theta }}J(\mathbf{\theta })$. The gradient is a vector of partial derivatives, $\left[\frac{\partial J}{\partial {\theta }_0},\frac{\partial J}{\partial {\theta }_1},\dots ,\frac{\partial J}{\partial {\theta }_m}\right]$, which points in the direction of the steepest ascent of the cost function. By moving in the opposite direction, we can iteratively update our parameters to find the minimum cost. This derivation is the mathematical core of the gradient descent optimization algorithm, which we will implement in the next lesson.

1. Essential Calculus: The Derivative of the Sigmoid Function

Before we derive the gradient of the entire cost function, let's derive a key intermediate result: the derivative of the sigmoid function, $\sigma (z)$. This beautiful and compact result is a cornerstone of the derivations for many neural network models, not just logistic regression.

Recall the sigmoid function:
$\sigma (z)=\frac{1}{1+e^{-z}}$

To find its derivative with respect to $z$, we can write it as $\sigma (z)=(1+e^{-z}{)}^{-1}$ and apply the chain rule.

1. Apply the power rule and chain rule:
   $\frac{d\sigma }{dz}=-1\cdot (1+e^{-z}{)}^{-2}\cdot \frac{d}{dz}(1+e^{-z})$
   $\frac{d\sigma }{dz}=-(1+e^{-z}{)}^{-2}\cdot (-e^{-z})=\frac{e^{-z}}{(1+e^{-z}{)}^2}$

2. Perform an algebraic manipulation: Now, we'll use a clever trick to express this result in terms of $\sigma (z)$ itself.
   $\frac{d\sigma }{dz}=\left(\frac{1}{1+e^{-z}}\right)\left(\frac{e^{-z}}{1+e^{-z}}\right)$
   Notice that the first term is simply $\sigma (z)$. Let's manipulate the second term:
   $\frac{e^{-z}}{1+e^{-z}}=\frac{(1+e^{-z})-1}{1+e^{-z}}=1-\frac{1}{1+e^{-z}}=1-\sigma (z)$

3. Combine the terms:
   $\frac{d\sigma }{dz}=\sigma (z)(1-\sigma (z))$
   This elegant result—that the derivative of the sigmoid function can be expressed in terms of the function's own output—is what makes the gradient calculation for logistic regression remarkably clean.

2. Deriving the Gradient with the Chain Rule

We are now ready to compute the partial derivative of the cost function $J(\mathbf{\theta })$ with respect to a single parameter ${\theta }_j$.
Because the derivative of a sum is the sum of derivatives, we can first focus on the cost contribution from a single training example $({\mathbf{x}}^{(i)},y^{(i)})$ and then sum the results. The cost for one example is:
$J_i(\mathbf{\theta })=-[y^{(i)}\log (h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))]$
We want to find $\frac{\partial J_i}{\partial {\theta }_j}$. This is a perfect application for the chain rule, as $J_i$ is a function of $h$, which is a function of $z$, which in turn is a function of ${\theta }_j$.
$\frac{\partial J_i}{\partial {\theta }_j}=\frac{\partial J_i}{\partial h}\cdot \frac{\partial h}{\partial z}\cdot \frac{\partial z}{\partial {\theta }_j}$
The following video provides a clear, step-by-step walkthrough of this exact process. It uses slightly different notation (e.g., L for loss, a for activation/hypothesis, w for weights), but the mathematical steps are identical.

Let's re-trace those steps here using our notation to solidify the derivation.

1. Compute $\frac{\partial J_i}{\partial h}$:
   $\frac{\partial J_i}{\partial h}=-\left[y^{(i)}\frac{1}{h}+(1-y^{(i)})\frac{1}{1-h}(-1)\right]=-\left[\frac{y^{(i)}}{h}-\frac{1-y^{(i)}}{1-h}\right]$
   $=\frac{-(y^{(i)}(1-h)-h(1-y^{(i)}))}{h(1-h)}=\frac{-(y^{(i)}-y^{(i)}h-h+h{y}^{(i)})}{h(1-h)}=\frac{h-y^{(i)}}{h(1-h)}$

2. Compute $\frac{\partial h}{\partial z}$:
   This is the sigmoid derivative we derived earlier. Since $h=\sigma (z)$, we have:
   $\frac{\partial h}{\partial z}=h(1-h)$

3. Compute $\frac{\partial z}{\partial {\theta }_j}$:
   The linear component is $z={\mathbf{\theta }}^T{\mathbf{x}}^{(i)}={\theta }_0{x}_0^{(i)}+{\theta }_1{x}_1^{(i)}+\cdots +{\theta }_j{x}_j^{(i)}+\dots$. The partial derivative with respect to ${\theta }_j$ is simply the coefficient of ${\theta }_j$:
   $\frac{\partial z}{\partial {\theta }_j}=x_j^{(i)}$

4. Combine the components:
   $\frac{\partial J_i}{\partial {\theta }_j}=\left(\frac{h-y^{(i)}}{h(1-h)}\right)\cdot \left(h(1-h)\right)\cdot \left(x_j^{(i)}\right)$
   The $h(1-h)$ terms cancel out beautifully, leaving us with a very simple result:
   $\frac{\partial J_i}{\partial {\theta }_j}=(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)})-y^{(i)})x_j^{(i)}$

This is the gradient of the cost for a single example. To get the gradient of the total average cost function $J(\mathbf{\theta })$, we sum over all $n$ examples and divide by $n$:
$\frac{\partial J(\mathbf{\theta })}{\partial {\theta }_j}=\frac{1}{n}{\sum }_{i=1}^n(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)})-y^{(i)})x_j^{(i)}$
For a formal written account of this derivation, the following resource is excellent.

3. A Surprising and Elegant Connection

Let's look at our final gradient expression again:
$\frac{\partial J(\mathbf{\theta })}{\partial {\theta }_j}=\frac{1}{n}{\sum }_{i=1}^n\underset{\text{Prediction Error}}{\underbrace{(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)})-y^{(i)})}}\cdot x_j^{(i)}$
This is the average of the prediction error for each data point, multiplied by its corresponding feature value. This form is remarkably intuitive.

What is more remarkable is its similarity to the gradient for linear regression. The gradient descent update rule for both models takes the form:
${\theta }_j:={\theta }_j-\alpha \cdot \text{gradient}$
If you recall from your work with linear regression, its gradient also has the form $\frac{1}{n}\sum (h(\mathbf{x})-y)x_j$. The update rules look identical!

The key insight is that while the form of the update rule is the same, the definition of the hypothesis $h_{\mathbf{\theta }}(\mathbf{x})$ is different:

• Linear Regression: $h_{\mathbf{\theta }}(\mathbf{x})={\mathbf{\theta }}^T\mathbf{x}$
• Logistic Regression: $h_{\mathbf{\theta }}(\mathbf{x})=\frac{1}{1+e^{-{\mathbf{\theta }}^T\mathbf{x}}}$

This elegant unification occurs because both linear and logistic regression are special cases of a broader class of algorithms called Generalized Linear Models (GLMs). This is a topic for another time, but it's a profound result that demonstrates a deep connection between seemingly different models.

Conclusion

Today, we successfully derived the gradient for the logistic regression cost function, a pivotal step in understanding how the model learns.

Key Takeaways:

• The derivative of the sigmoid function is ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$, a result which simplifies the gradient derivation.
• Using the chain rule, we decomposed the complex derivative of the cost function into three simpler parts.
• The partial derivative of the average cross-entropy cost function with respect to a parameter ${\theta }_j$ is:
  $\frac{\partial J(\mathbf{\theta })}{\partial {\theta }_j}=\frac{1}{n}{\sum }_{i=1}^n(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)})-y^{(i)})x_j^{(i)}$
• The logistic regression gradient update rule has the same form as the linear regression rule, differing only in the definition of the hypothesis function $h_{\mathbf{\theta }}(\mathbf{x})$.

Preview of the Next Lesson:
With the cost function $J(\mathbf{\theta })$ and its gradient ${\nabla }_{\mathbf{\theta }}J(\mathbf{\theta })$ in hand, we have all the components needed for optimization. In our next lesson, we will apply the gradient descent algorithm to iteratively update the parameters $\mathbf{\theta }$ and train a logistic regression model on a dataset.