Key Estimation Methods Explained

1. Method of Moments (MoM): Matching Sample to Theory

The Method of Moments is perhaps the most straightforward way to devise an estimator. The core idea is simple and intuitive: we assume that the moments calculated from our sample should be good approximations of the theoretical moments of the population's underlying distribution.

A population moment is a theoretical property of a distribution. The k-th population moment is defined as ${\mu }_k^{\prime }=E[X^k]$. These moments are often functions of the distribution's unknown parameters, $\theta$.
A sample moment is a value we can compute from our data. The k-th sample moment is $m_k^{\prime }=\frac{1}{n}{\sum }_{i=1}^n{X}_i^k$.

The MoM strategy is to:

1. Express the first few population moments (${\mu }_1^{\prime },{\mu }_2^{\prime },\dots$) in terms of the unknown parameters (${\theta }_1,{\theta }_2,\dots$).
2. Set these theoretical expressions equal to their corresponding sample moments ($m_1^{\prime },m_2^{\prime },\dots$).
3. Solve the resulting system of equations for the unknown parameters.

The following video provides an excellent, clear introduction to this idea and works through two key examples.

As the video demonstrated, for a normal distribution $N(\mu ,{\sigma }^2)$, we have two parameters, so we need two moments:

1. First moment: $E[X]=\mu$. We set this equal to the first sample moment, $m_1^{\prime }=\bar{X}$. This immediately gives us our estimator: ${\hat{\mu }}_{MoM}=\bar{X}$.
2. Second moment: $E[X^2]=Var(X)+(E[X]{)}^2={\sigma }^2+{\mu }^2$. We set this equal to the second sample moment, $m_2^{\prime }=\frac{1}{n}\sum X_i^2$.
   $\frac{1}{n}\sum X_i^2={\hat{\sigma }}^2+{\hat{\mu }}^2$
   Solving for ${\hat{\sigma }}^2$ and substituting $\hat{\mu }=\bar{X}$ gives:
   ${\hat{\sigma }}_{MoM}^2=\frac{1}{n}\sum X_i^2-{\bar{X}}^2=\frac{1}{n}\sum (X_i-\bar{X}{)}^2$

The following reading provides a more formal definition and discusses the pros and cons of this method.

Purpose and Summary of MoM:

• Purpose: To provide a quick, simple, and intuitive method for finding parameter estimates.
• Pros: Easy to understand and apply. Estimators are often consistent.
• Cons: Can be less efficient (higher variance) than other methods. Sometimes, the estimates can be nonsensical (e.g., a negative estimate for a variance parameter). It's often considered a "quick and dirty" starting point.

2. Maximum Likelihood Estimation (MLE): Maximizing Plausibility

Maximum Likelihood Estimation is a more foundational and powerful method that dominates modern statistics and machine learning. Its guiding principle is: we should choose the parameter values that make our observed data the most probable outcome.

Let's say we have a sample of i.i.d. data points $x_1,x_2,\dots ,x_n$ from a distribution $f(x|\theta )$. The joint probability of observing this specific dataset is the product of the individual probabilities (or densities):
$P(x_1,\dots ,x_n|\theta )={\prod }_{i=1}^{n}f(x_i|\theta )$
When we view this expression as a function of the parameter $\theta$ for our fixed, observed data, we call it the likelihood function, $L(\theta |x_1,\dots ,x_n)$.

The MLE is the value of $\theta$ that maximizes this function.

Because maximizing a product of terms can be mathematically cumbersome, we almost always work with the log-likelihood function, $\ln L(\theta |x)$. Since the logarithm is a monotonically increasing function, maximizing the log-likelihood is equivalent to maximizing the likelihood itself. The logarithm handily turns the product into a sum:
$\ln L(\theta |x)={\sum }_{i=1}^n\ln f(x_i|\theta )$
Finding the MLE then becomes a standard optimization problem, typically solved by taking the derivative with respect to $\theta$, setting it to zero, and solving.

Let's watch a video that brings this process to life.

Purpose and Summary of MLE:

• Purpose: To find the parameter values that best explain the observed data from a probabilistic standpoint.
• Pros: Deeply principled, applicable to almost any model. MLEs possess excellent large-sample properties: they are typically consistent, asymptotically unbiased, and have the lowest possible variance among a wide class of estimators.
• Cons: Can be more mathematically involved than MoM. For complex models, the optimization may need to be done numerically.

3. Bayesian Estimation: Updating Beliefs

The Bayesian approach represents a fundamental philosophical shift. In both MoM and MLE (known as frequentist methods), the parameter $\theta$ is considered a fixed, unknown constant. Our goal is to find a single best guess, or point estimate, for it.

In Bayesian inference, the parameter $\theta$ is itself considered a random variable. We begin with a prior distribution, $\pi (\theta )$, which represents our beliefs about $\theta$ before seeing any data. We then use the observed data to update our beliefs, resulting in a posterior distribution, $\pi (\theta |x)$.

This update is performed using Bayes' Rule:

$\pi (\theta |x)=\frac{P(x|\theta )\pi (\theta )}{P(x)}⟹\text{Posterior}=\frac{\text{Likelihood}\times \text{Prior}}{\text{Evidence}}$

Since the denominator, $P(x)$, does not depend on $\theta$, it acts as a normalization constant. We can therefore state the core of Bayesian inference as:

$\text{Posterior}\propto \text{Likelihood}\times \text{Prior}$

Unlike MoM and MLE, the result of a Bayesian estimation is not a single point but an entire probability distribution for the parameter. This distribution quantifies our uncertainty about $\theta$ after observing the data. From this posterior, we can compute a point estimate if needed (e.g., the mean, median, or mode of the posterior), but we also get a measure of confidence (e.g., the variance or a credible interval).

To solidify these concepts, the following reading provides a fantastic comparison of all three estimation philosophies.

Purpose and Summary of Bayesian Estimation:

• Purpose: To formally update prior beliefs about parameters using observed data, yielding a full probability distribution that quantifies our posterior uncertainty.
• Pros: Provides a complete distributional description of the parameter. Allows for the formal incorporation of prior knowledge, which can be very powerful with small datasets.
• Cons: Can be computationally intensive, as it may require complex integration to find the normalization constant. The choice of prior can sometimes be subjective.

Conclusion

We have now surveyed the three principal methods for constructing estimators. They represent different philosophical approaches to the same fundamental problem: learning from data.

Key Takeaways:

Understanding these three approaches is crucial. While MLE is arguably the most common technique in frequentist statistics and machine learning, the principles of MoM are valuable for their simplicity, and the Bayesian framework is essential for modern applications involving uncertainty quantification.

Preview of the Next Lesson:

Now that we understand the what and why of these methods, we can focus on the how. In the next lesson, we will get our hands dirty with the mathematics of Maximum Likelihood Estimation. We will perform detailed derivations of the MLEs for several common distributions, including the Normal, Exponential, and Poisson. We will also derive the Ordinary Least Squares (OLS) estimator for linear regression and discover its deep connection to MLE, fulfilling your request for rigorous mathematical proofs and connections between concepts.