Log-Likelihood for Logistic Regression

Hello! Let's continue our exploration of logistic regression.

In our last lesson, we established the probabilistic foundation of the model. We saw that logistic regression assumes each binary label $y^{(i)}$ follows a Bernoulli distribution, and we constructed the overall likelihood function $L(\mathbf{\theta })$ by taking the product of the probabilities for each data point. This function, $L(\mathbf{\theta })=\prod P(y^{(i)}|{\mathbf{x}}^{(i)};\mathbf{\theta })$, represents the probability of observing our entire dataset for a given set of parameters $\mathbf{\theta }$.

Today, we will address the learning outcome: to derive the log-likelihood function (cross-entropy loss) for logistic regression. We will transform the likelihood function into a more mathematically tractable form, the log-likelihood, and then reframe our optimization goal from maximization to minimization, which will lead us directly to one of the most important cost functions in machine learning.

1. From Products to Sums: The Need for Logarithms

Our current objective is to find the parameters $\mathbf{\theta }$ that maximize the likelihood function:
$L(\mathbf{\theta })={\prod }_{i=1}^n(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{y^{(i)}}(1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{1-y^{(i)}}$

Attempting to maximize this function directly presents two major challenges:

1. Computational Instability: The term $h_{\mathbf{\theta }}({\mathbf{x}}^{(i)})$ is a probability, a value between 0 and 1. Multiplying many such numbers together results in an extremely small value that can easily become zero due to floating-point precision limitations in computers (a problem known as numerical underflow). This makes comparing the likelihood of different $\mathbf{\theta }$ values impossible.
2. Mathematical Inconvenience: Finding the maximum of a function typically involves taking its derivative and setting it to zero. The derivative of a long product is very complex to compute (requiring repeated application of the product rule). In contrast, the derivative of a sum is simple.

To solve both problems, we can work with the logarithm of the likelihood instead. The natural logarithm, $\log (x)$, is a strictly monotonically increasing function. This means that if $a>b$, then $\log (a)>\log (b)$. Consequently, the value of $\mathbf{\theta }$ that maximizes $L(\mathbf{\theta })$ will also be the value that maximizes $\log (L(\mathbf{\theta }))$.

The following video provides an excellent intuitive explanation of this transition.

2. Formal Derivation of the Log-Likelihood Function

Now, let's perform the formal derivation. We define the log-likelihood function, denoted $\ell (\mathbf{\theta })$, as the natural logarithm of the likelihood function $L(\mathbf{\theta })$.

$\ell (\mathbf{\theta })=\log L(\mathbf{\theta })=\log \left[{\prod }_{i=1}^n(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{y^{(i)}}(1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{1-y^{(i)}}\right]$

Using the fundamental properties of logarithms, specifically $\log (ab)=\log (a)+\log (b)$ and $\log (a^k)=k\log (a)$, we can simplify this expression.

1. The logarithm of a product becomes the sum of the logarithms:
   $\ell (\mathbf{\theta })={\sum }_{i=1}^n\log \left[(h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{y^{(i)}}(1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{1-y^{(i)}}\right]$

2. The logarithm of a product within the sum becomes a sum of logarithms:
   $\ell (\mathbf{\theta })={\sum }_{i=1}^n\left[\log \left((h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{y^{(i)}}\right)+\log \left((1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}){)}^{1-y^{(i)}}\right)\right]$

3. The exponents can be brought down as multipliers:
   $\ell (\mathbf{\theta })={\sum }_{i=1}^n\left[y^{(i)}\log (h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))\right]$

This final expression is the log-likelihood function for logistic regression. Our goal is now to find the parameters $\mathbf{\theta }$ that maximize this sum, a much more manageable task.

For a rigorous, academic presentation of this derivation, please refer to the following text.

3. From Maximization to Minimization: The Binary Cross-Entropy Loss

By convention, most optimization algorithms in machine learning, including gradient descent, are formulated as minimizing a cost function $J(\mathbf{\theta })$ (also called a loss or error function).

Maximizing a function is equivalent to minimizing its negative. Therefore, we can reframe our goal of maximizing the log-likelihood $\ell (\mathbf{\theta })$ as minimizing the negative log-likelihood, $-\ell (\mathbf{\theta })$.

We define our cost function $J(\mathbf{\theta })$ as the average negative log-likelihood over the training set. We average by dividing by the number of training examples, $n$, which makes the cost independent of the dataset size but does not change the location of the minimum.

$J(\mathbf{\theta })=-\frac{1}{n}\ell (\mathbf{\theta })=-\frac{1}{n}{\sum }_{i=1}^n\left[y^{(i)}\log (h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))\right]$

This function, $J(\mathbf{\theta })$, is the canonical cost function for logistic regression. It has a specific name: Binary Cross-Entropy Loss, or sometimes Log Loss.

Let's examine the contribution of a single data point $(\mathbf{x},y)$ to this cost:
Cost(h_θ(x), y) = -[y log(h_θ(x)) + (1-y) log(1-h_θ(x))]

• If the true label is $y=1$, the cost becomes -log(h_θ(x)). As our prediction h_θ(x) approaches 1 (correct), the cost -log(h_θ(x)) approaches 0. As h_θ(x) approaches 0 (incorrect), the cost -log(h_θ(x)) approaches $\infty$. The model is heavily penalized for being confidently wrong.
• If the true label is $y=0$, the cost becomes -log(1 - h_θ(x)). As our prediction h_θ(x) approaches 0 (correct), 1-h_θ(x) approaches 1, and the cost -log(1-h_θ(x)) approaches 0. As h_θ(x) approaches 1 (incorrect), 1-h_θ(x) approaches 0, and the cost -log(1-h_θ(x)) approaches $\infty$.

The following resource provides a clear derivation of this final cost function and its interpretation.

Conclusion

Today we made the crucial transition from a probabilistic statement (the likelihood) to a concrete optimization objective (the cost function).

Key Takeaways:

• Maximizing the likelihood function $L(\mathbf{\theta })$ is problematic due to numerical underflow and mathematical complexity.
• Since the logarithm is a monotonic function, maximizing $L(\mathbf{\theta })$ is equivalent to maximizing the log-likelihood function $\ell (\mathbf{\theta })$.
• Taking the logarithm transforms the product over data points into a more manageable sum.
• By convention, we frame optimization as a minimization problem. Therefore, we minimize the negative log-likelihood.
• The final cost function for logistic regression is the average negative log-likelihood, known as the Binary Cross-Entropy Loss:
  $J(\mathbf{\theta })=-\frac{1}{n}{\sum }_{i=1}^n\left[y^{(i)}\log (h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))+(1-y^{(i)})\log (1-h_{\mathbf{\theta }}({\mathbf{x}}^{(i)}))\right]$

Preview of the Next Lesson:
We have successfully derived the cost function $J(\mathbf{\theta })$ that we need to minimize. The next logical step is to find the values of $\mathbf{\theta }$ that achieve this minimum. In the next lesson, we will derive the gradient of this objective function, ${\nabla }_{\mathbf{\theta }}J(\mathbf{\theta })$. This gradient will be the engine that powers the gradient descent algorithm, allowing us to iteratively update our parameters and "learn" the optimal logistic regression model.