Log-Likelihood for GLMs

From Probability to Likelihood

The idea of likelihood is a subtle but powerful shift in perspective from probability.

• Probability asks: Given a model with known parameters, what is the chance of seeing a particular outcome?
• Likelihood asks: Given the data we have actually observed, how plausible is a particular set of model parameters?

For a set of independent observations $y_1,y_2,\dots ,y_n$, the total likelihood is the product of the individual probabilities. However, working with products is mathematically inconvenient. This is why we almost always work with the log-likelihood.

The following video provides an excellent introduction to Maximum Likelihood Estimation and explains why the log-likelihood is so useful.

As the video highlights, using the log-likelihood simplifies our problem immensely by converting a large product of probabilities into a more manageable sum. Maximizing $\log (L)$ is equivalent to maximizing $L$ because the logarithm is a monotonically increasing function.

The Log-Likelihood for the Exponential Family

Now, let's generalize this. In the previous module, you saw that any distribution in the exponential family can be written in a standard form. This standardization allows us to define a single, general formula for the log-likelihood of any GLM.

Let's use the general form we've seen before. The probability density/mass function (PDF/PMF) for a single observation $y_i$ from an exponential family distribution is:

Here, ${\theta }_i$ is the natural parameter and $\varphi$ is the dispersion parameter.

To get the log-likelihood for this single observation, we simply take the natural logarithm.

As shown in the resource, taking the log of the exponential function simply gives us the exponent. The log-likelihood for a single observation $y_i$ is:

Since our observations are independent, the total log-likelihood for the entire dataset is the sum of these individual log-likelihoods:

From $\theta$ to $\beta$: Making it a GLM

The expression above is a function of the natural parameters ${\theta }_i$. However, in a regression context, our goal is to estimate the coefficients $\mathbf{\beta }$. We need to express the log-likelihood as a function of $\mathbf{\beta }$.

This connection is made through the components of the GLM you've already learned:

1. The linear predictor defines ${\eta }_i$ as a linear combination of the predictors and coefficients: ${\eta }_i={\mathbf{x}}_i^T\mathbf{\beta }$.
2. The link function connects the mean ${\mu }_i$ to the linear predictor: $g({\mu }_i)={\eta }_i$.
3. A property of the exponential family connects the mean ${\mu }_i$ to the natural parameter ${\theta }_i$ via the derivative of $b({\theta }_i)$: ${\mu }_i=b^{\prime }({\theta }_i)$.

Therefore, the parameters we want to find, $\mathbf{\beta }$, determine the log-likelihood through this chain:

The function we ultimately want to maximize is $l(\mathbf{\beta })$.

Worked Example: The Poisson GLM

Let's make this concrete by formulating the log-likelihood for a Poisson GLM.

1. Start with the PMF: The probability of observing a count $y_i$ from a Poisson distribution with mean ${\mu }_i$ is:

   $$f(y_i|{\mu }_i)=\frac{e^{-{\mu }_i}{\mu }_i^{y_i}}{y_i!}$$

2. Take the Log: The log-likelihood for a single observation is:

   $$l_i({\mu }_i|y_i)=\log \left(\frac{e^{-{\mu }_i}{\mu }_i^{y_i}}{y_i!}\right)=y_i\log ({\mu }_i)-{\mu }_i-\log (y_i!)$$

3. Sum over all observations: The total log-likelihood as a function of the means ${\mu }_i$ is:

   $$l(\mathbf{\mu }|\mathbf{y})=\sum _{i=1}^n\left(y_i\log ({\mu }_i)-{\mu }_i-\log (y_i!)\right)$$

4. Introduce the GLM link function: For a Poisson GLM, the canonical link is the log link.

   $$g({\mu }_i)=\log ({\mu }_i)={\eta }_i={\mathbf{x}}_i^T\mathbf{\beta }$$

   This implies that ${\mu }_i=\exp ({\eta }_i)=\exp ({\mathbf{x}}_i^T\mathbf{\beta })$.

5. Substitute to get the final log-likelihood in terms of $\mathbf{\beta }$: We replace $\log ({\mu }_i)$ with ${\mathbf{x}}_i^T\mathbf{\beta }$ and ${\mu }_i$ with $\exp ({\mathbf{x}}_i^T\mathbf{\beta })$.

   $$l(\mathbf{\beta }|\mathbf{y})=\sum _{i=1}^n\left(y_i({\mathbf{x}}_i^T\mathbf{\beta })-\exp ({\mathbf{x}}_i^T\mathbf{\beta })-\log (y_i!)\right)$$

This is the function that an optimization algorithm would maximize to find the MLEs for the coefficients $\mathbf{\beta }$ in a Poisson regression.

Practice: Formulating the Logistic Regression Log-Likelihood

Now it's your turn. Logistic regression is a GLM for binary data, using a Binomial distribution with $n=1$ (also called a Bernoulli distribution) and a logit link. Follow the steps above to formulate its log-likelihood function.

Conclusion

In this lesson, you have learned the crucial process of building the objective function for parameter estimation in GLMs. This bridges the gap between choosing a distribution and actually fitting the model.

Key Takeaways:

• Maximum Likelihood Estimation (MLE) seeks to find the parameters that make the observed data most probable.
• We work with the log-likelihood because it turns products into sums, simplifying the mathematics of optimization.
• The log-likelihood for a GLM starts from the general exponential family form.
• The final log-likelihood function must be expressed in terms of the regression coefficients $\mathbf{\beta }$ by substituting the model's link function.

You now know how to construct the function $l(\mathbf{\beta })$. The next natural question is: how do we find the $\mathbf{\beta }$ that maximizes it? In the next lesson, you will learn how to take the first step in this optimization process by deriving the score equations, which are simply the derivatives of the log-likelihood function.