MLE for Common Distributions

1. The Principle of Maximum Likelihood

Before we start deriving estimators, let's formally define the core concepts. We need to distinguish between a probability distribution and a likelihood function, and understand why we almost always work with the log-likelihood.

To summarize the key ideas from the reading:

• Likelihood Function, L(θ|x): For an observed dataset $\mathbf{x}=(x_1,...,x_n)$, the likelihood function is the joint probability density (or mass) function $f(\mathbf{x};\theta )$ treated as a function of the parameter $\theta$. For i.i.d. data, this is $L(\theta |\mathbf{x})={\prod }_{i=1}^{n}f(x_i;\theta )$.
• Maximum Likelihood Estimator (MLE), ${\hat{\theta }}_{MLE}$: The MLE is the value of $\theta$ from the parameter space that maximizes the likelihood function. $${\hat{\theta }}_{MLE}=\arg \underset{\theta }{\max }L(\theta |\mathbf{x})$$
• Log-Likelihood Function, l(θ|x): To make the math easier, we maximize the natural logarithm of the likelihood, $l(\theta |\mathbf{x})=\ln L(\theta |\mathbf{x})$. Since log is a monotonically increasing function, this gives the same result. For i.i.d. data, this conveniently turns the product into a sum: $$l(\theta |\mathbf{x})=\sum _{i=1}^n\ln f(x_i;\theta )$$
• The Procedure:
  1. Write the log-likelihood function.
  2. Take its derivative with respect to the parameter $\theta$. This derivative is called the score function.
  3. Set the score function to zero and solve for $\theta$.
  4. Verify that the solution is a maximum (usually by checking that the second derivative is negative).

Now, let's apply this procedure to our three target distributions.

2. MLE for the Normal Distribution

We'll start with the most common continuous distribution, the Normal distribution, which has two parameters: the mean $\mu$ and the variance ${\sigma }^2$. The derivation involves partial derivatives with respect to both parameters.

The following video provides an exceptionally clear, step-by-step walkthrough of this derivation.

As the video proved, for a sample $x_1,...,x_n$ from a Normal distribution $\mathcal{N}(\mu ,{\sigma }^2)$:

1. The log-likelihood function is:

   $$l(\mu ,{\sigma }^2|\mathbf{x})=-\frac{n}{2}\ln (2\pi )-\frac{n}{2}\ln ({\sigma }^2)-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(x_i-\mu {)}^2$$

2. Taking the partial derivative with respect to $\mu$ and setting to zero gives the MLE for the mean:

   $$\frac{\partial l}{\partial \mu }=\frac{1}{{\sigma }^2}\sum _{i=1}^n(x_i-\mu )=0⟹\sum x_i-n\mu =0⟹{\hat{\mu }}_{MLE}=\frac{1}{n}\sum _{i=1}^n{x}_i=\bar{x}$$

3. Taking the partial derivative with respect to ${\sigma }^2$ and setting to zero gives the MLE for the variance:

   $$\frac{\partial l}{\partial {\sigma }^2}=-\frac{n}{2{\sigma }^2}+\frac{1}{2({\sigma }^2{)}^2}\sum _{i=1}^n(x_i-\mu {)}^2=0⟹{\hat{\sigma }}_{MLE}^2=\frac{1}{n}\sum _{i=1}^n(x_i-{\hat{\mu }}_{MLE}{)}^2$$

The MLE for the mean is the sample mean, and the MLE for the variance is the sample variance with a denominator of $n$.

3. MLE for the Exponential Distribution

Next, let's tackle the Exponential distribution, which models the time between events in a Poisson process. This is a simpler, single-parameter case.

As derived in the video and in Example 6.2.3 of the textbook (LINK), for a sample from an Exponential distribution with rate parameter $\lambda$, the PDF is $f(x;\lambda )=\lambda e^{-\lambda x}$.

1. The log-likelihood function is:

   $$l(\lambda |\mathbf{x})=\sum _{i=1}^n\ln (\lambda e^{-\lambda x_i})=\sum _{i=1}^n(\ln \lambda -\lambda x_i)=n\ln \lambda -\lambda \sum _{i=1}^n{x}_i$$

2. Taking the derivative with respect to $\lambda$ and setting it to zero gives:

   $$\frac{dl}{d\lambda }=\frac{n}{\lambda }-\sum _{i=1}^n{x}_i=0⟹{\hat{\lambda }}_{MLE}=\frac{n}{{\sum }_{i=1}^n{x}_i}=\frac{1}{\bar{x}}$$

The MLE for the rate parameter $\lambda$ is the reciprocal of the sample mean. This is intuitive: if the average time between events ($\bar{x}$) is long, the rate of events ($\lambda$) is low.

4. MLE for the Poisson Distribution

Finally, let's derive the MLE for a discrete distribution: the Poisson. This distribution models the number of events occurring in a fixed interval of time or space.

We'll work through this derivation together, applying the principles we've just learned. This task corresponds to Exercise 6.2.4 in the textbook (LINK).

Let $x_1,...,x_n$ be an i.i.d. sample from a Poisson($\lambda$) distribution. The probability mass function (PMF) is:

1. Write the likelihood function. The joint probability of observing our sample is:

   $$L(\lambda |\mathbf{x})=\prod _{i=1}^n\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}$$

2. Write the log-likelihood function.

   $$l(\lambda |\mathbf{x})=\ln \left(\prod _{i=1}^n\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}\right)=\sum _{i=1}^n\ln \left(\frac{e^{-\lambda }{\lambda }^{x_i}}{x_i!}\right)$$

   Using logarithm rules to simplify:

   $$l(\lambda |\mathbf{x})=\sum _{i=1}^n(\ln (e^{-\lambda })+\ln ({\lambda }^{x_i})-\ln (x_i!))$$ $$l(\lambda |\mathbf{x})=\sum _{i=1}^n(-\lambda +x_i\ln \lambda -\ln (x_i!))$$

   Summing over the $n$ terms:

   $$l(\lambda |\mathbf{x})=-n\lambda +(\ln \lambda )\sum _{i=1}^n{x}_i-\sum _{i=1}^n\ln (x_i!)$$

3. Take the derivative with respect to $\lambda$. The last term, $\sum \ln (x_i!)$, does not depend on $\lambda$, so its derivative is zero.

   $$\frac{dl}{d\lambda }=-n+\frac{1}{\lambda }\sum _{i=1}^n{x}_i-0$$

4. Set the derivative to zero and solve.

   $$-n+\frac{1}{\lambda }\sum _{i=1}^n{x}_i=0⟹\frac{1}{\lambda }\sum _{i=1}^n{x}_i=n$$ $${\hat{\lambda }}_{MLE}=\frac{{\sum }_{i=1}^n{x}_i}{n}=\bar{x}$$

The Maximum Likelihood Estimator for the rate parameter $\lambda$ of a Poisson distribution is simply the sample mean.

Conclusion

In this lesson, we established a general and powerful principle for finding estimators. By maximizing the likelihood (or log-likelihood) function, we can derive estimators for the parameters of a wide variety of statistical models.

Key Takeaways:

• Maximum Likelihood Estimation (MLE) finds the parameter values that make the observed data most probable.
• The log-likelihood function is a critical computational tool that simplifies the optimization problem by converting products into sums.
• We derived the MLEs for three key distributions:
  • Normal $\mathcal{N}(\mu ,{\sigma }^2)$: ${\hat{\mu }}_{MLE}=\bar{x}$ and ${\hat{\sigma }}_{MLE}^2=\frac{1}{n}\sum (x_i-\bar{x}{)}^2$.
  • Exponential($\lambda$): ${\hat{\lambda }}_{MLE}=1/\bar{x}$.
  • Poisson($\lambda$): ${\hat{\lambda }}_{MLE}=\bar{x}$.

You may notice that these estimators are often functions of the sufficient statistics we identified in the previous lesson (e.g., $\sum x_i$ and $\sum x_i^2$). This is a deep and important property of MLEs that we will revisit.

Self-Test

Let's test your understanding of the MLE procedure. Consider a random sample of size $n$, $X_1,...,X_n$, from a Bernoulli($p$) distribution, where $p$ is the unknown probability of success. The PMF is $P(X=k;p)=p^k(1-p{)}^{1-k}$ for $k\in \{0,1\}$.

1. Write down the log-likelihood function $l(p|\mathbf{x})$.
2. Take the derivative of the log-likelihood with respect to $p$.
3. Set the derivative to zero and solve to find the MLE for $p$. Does the result seem intuitive?

Preview of the Next Lesson:

We have now seen how to derive estimators for the parameters of specific probability distributions. In the next lesson, we will shift our focus to a different but related problem: modeling the relationship between variables. We will introduce Ordinary Least Squares (OLS) and derive the famous Normal Equations using matrix calculus. We will then discover a profound connection: for a linear model with normally distributed errors, the OLS estimator is identical to the Maximum Likelihood Estimator. This will bridge the gap between probabilistic estimation and linear regression.