Understanding the Logistic Regression Decision Boundary

Hello! Welcome to the final lesson in our module on logistic regression.

In our last session, we built the engine for our model by implementing the gradient descent algorithm. We saw how to iteratively update the parameter vector $\mathbf{\theta }$ by moving in the direction opposite to the gradient of the cost function, ensuring we find the optimal parameters that minimize prediction error.

Now that we know how to find the best $\mathbf{\theta }$, a crucial question remains: what do these learned parameters actually represent? This lesson will connect the abstract numbers in the $\mathbf{\theta }$ vector to a concrete, geometric concept. We will explore how these parameters define the decision boundary, the line (or plane) that the model uses to separate the different classes.

By the end of this lesson, you will be able to define the decision boundary for binary logistic regression and explain precisely how the learned parameters determine its characteristics.

1. Defining the Decision Boundary

In binary logistic regression, our model, $h_{\mathbf{\theta }}(\mathbf{x})$, outputs the estimated probability that an input $\mathbf{x}$ belongs to the positive class ($y=1$). The classification rule is straightforward:

• If $h_{\mathbf{\theta }}(\mathbf{x})\ge 0.5$, we predict $y=1$.
• If $h_{\mathbf{\theta }}(\mathbf{x})<0.5$, we predict $y=0$.

The decision boundary is the threshold where the model is perfectly uncertain. It's the set of all points $\mathbf{x}$ for which the predicted probability is exactly 0.5.

Let's derive the equation for this boundary. We start with our model's hypothesis:

We set this probability to 0.5:

Solving for the exponent:
\begin{align*}
1 &= 0.5 (1 + e^{-\boldsymbol{\theta}^T \mathbf{x}}) \
2 &= 1 + e^{-\boldsymbol{\theta}^T \mathbf{x}} \
1 &= e^{-\boldsymbol{\theta}^T \mathbf{x}}
\end{align*}
Taking the natural logarithm of both sides:
\begin{align*}
\ln(1) &= \ln(e^{-\boldsymbol{\theta}^T \mathbf{x}}) \
0 &= -\boldsymbol{\theta}^T \mathbf{x} \
\boldsymbol{\theta}^T \mathbf{x} &= 0
\end{align*}

This simple equation, ${\mathbf{\theta }}^T\mathbf{x}=0$, is the mathematical definition of the decision boundary in logistic regression. All points $\mathbf{x}$ in the feature space that satisfy this equation lie on the boundary.

This is also highlighted in the following resource.

2. The Geometry of the Decision Boundary

The equation ${\mathbf{\theta }}^T\mathbf{x}=0$ defines a hyperplane. A hyperplane is a generalization of a line (in 2D) or a plane (in 3D) to any number of dimensions. Let's make this concrete for a 2D case where we have two features, $x_1$ and $x_2$.

Our feature vector is $\mathbf{x}=[1,x_1,x_2{]}^T$ (including the intercept term) and our parameter vector is $\mathbf{\theta }=[{\theta }_0,{\theta }_1,{\theta }_2{]}^T$. The decision boundary equation becomes:

This is the equation of a straight line in the $x_1,x_2$ plane. For any point on this line, the model outputs $P(y=1|\mathbf{x})=0.5$. For points on one side of the line, the probability will be $>0.5$ (predict 1), and for points on the other side, it will be $<0.5$ (predict 0).

The image below provides an excellent visualization.

3. How Parameters Determine the Boundary

Now we'll dissect how each part of the parameter vector $\mathbf{\theta }$ shapes this boundary. The following video provides a clear and intuitive explanation of these roles.

Let's summarize and formalize the insights from the video. We'll separate the bias/intercept term ${\theta }_0$ from the rest of the weights $\mathbf{w}=[{\theta }_1,\dots ,{\theta }_d{]}^T$. The boundary equation is ${\theta }_0+{\mathbf{w}}^T{\mathbf{x}}_{\mathrm{1..}d}=0$.

The Weight Vector $\mathbf{w}$: Orientation

The weight vector $\mathbf{w}$ determines the orientation or slope of the decision boundary.

A key mathematical property, as proven in the video, is that the vector $\mathbf{w}$ is always orthogonal (perpendicular) to the decision boundary.

Let's think about why this is true. Consider two points ${\mathbf{x}}_A$ and ${\mathbf{x}}_B$ that both lie on the decision boundary. Then:

1. ${\theta }_0+{\mathbf{w}}^T{\mathbf{x}}_A=0$
2. ${\theta }_0+{\mathbf{w}}^T{\mathbf{x}}_B=0$

Subtracting the two equations gives:

The vector $({\mathbf{x}}_A-{\mathbf{x}}_B)$ is a vector that lies in the hyperplane of the decision boundary (it connects two points on the boundary). The equation ${\mathbf{w}}^T({\mathbf{x}}_A-{\mathbf{x}}_B)=0$ means that the dot product of $\mathbf{w}$ and any vector in the decision boundary hyperplane is zero. By definition, this means $\mathbf{w}$ is orthogonal to the hyperplane.

The vector $\mathbf{w}$ points in the direction of the steepest increase in probability.

The Bias Term ${\theta }_0$: Position

The bias term ${\theta }_0$ determines the position or offset of the boundary. It shifts the boundary away from the origin.

If ${\theta }_0=0$, the equation becomes ${\mathbf{w}}^T\mathbf{x}=0$, which defines a hyperplane that passes through the origin. A non-zero ${\theta }_0$ shifts this hyperplane parallel to itself. The magnitude and sign of ${\theta }_0$ determine how far and in which direction (relative to $\mathbf{w}$) it is shifted.

The Magnitude of $\mathbf{w}$: Confidence

The magnitude or norm of the weight vector, $||\mathbf{w}||$, determines the steepness of the probability transition across the decision boundary.

• A large $||\mathbf{w}||$ means that ${\mathbf{w}}^T\mathbf{x}$ changes rapidly as $\mathbf{x}$ moves away from the boundary. This causes the sigmoid function to transition very sharply from 0 to 1. The model becomes very "confident" in its predictions, even for points very close to the boundary.
• A small $||\mathbf{w}||$ results in a more gradual change in ${\mathbf{w}}^T\mathbf{x}$ and a slower transition in the sigmoid function. The probabilities change more smoothly, and the model is less confident in its predictions near the boundary.

This is a crucial concept when we later discuss regularization, where we often penalize large weights to prevent overfitting and overly confident models.

Conclusion

In this lesson, we demystified the parameters learned by a logistic regression model by giving them a clear geometric interpretation. We have established that for binary classification, these parameters define a linear decision boundary.

Key Takeaways:

• The decision boundary is the set of points where the model's predicted probability is exactly 0.5. Its equation is ${\mathbf{\theta }}^T\mathbf{x}=0$.
• In a 2D feature space, this boundary is a straight line. For $d$ features, it is a $(d-1)$-dimensional hyperplane.
• The weight vector $\mathbf{w}$ (parameters ${\theta }_1,\dots ,{\theta }_d$) is orthogonal to the decision boundary and determines its orientation.
• The bias term ${\theta }_0$ determines the position or offset of the boundary from the origin.
• The magnitude of the weight vector, $||\mathbf{w}||$, controls the steepness of the probability curve, which translates to the model's prediction confidence.

This concludes our deep dive into logistic regression. You now understand its probabilistic formulation, how to derive its cost function and gradient, how to optimize it with gradient descent, and what the resulting model represents geometrically.

Preview of the Next Module:
In the next module, we will explore another powerful linear classifier: the Support Vector Machine (SVM). While an SVM also uses a decision boundary to separate classes, we will see that it chooses the 'best' boundary based on a very different and elegant principle: maximizing the margin, or the 'street,' between the closest points of the opposing classes. This will provide a fascinating contrast to the probabilistic approach of logistic regression.