Invariance of MLEs

1. The Principle of Invariance

The invariance property of Maximum Likelihood Estimators is straightforward to state:

If ${\hat{\theta }}_{MLE}$ is the Maximum Likelihood Estimate of a parameter $\theta$, and $g(\theta )$ is some function of $\theta$, then the Maximum Likelihood Estimate of $g(\theta )$ is simply $g({\hat{\theta }}_{MLE})$.

Let's make this concrete with a simple example you're already familiar with. For i.i.d. data from a Normal distribution $N(\mu ,{\sigma }^2)$, we found the MLEs to be:
${\hat{\mu }}_{MLE}=\bar{x}\text{and}{\hat{\sigma }}_{MLE}^2=\frac{1}{n}{\sum }_{i=1}^n(x_i-\bar{x}{)}^2$
Now, suppose we are interested in the standard deviation, $\sigma$. This is a function of the variance: $\sigma =g({\sigma }^2)=\sqrt{{\sigma }^2}$. According to the invariance property, the MLE for $\sigma$ is:
${\hat{\sigma }}_{MLE}=g({\hat{\sigma }}_{MLE}^2)=\sqrt{{\hat{\sigma }}_{MLE}^2}=\sqrt{\frac{1}{n}{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}$
This result is intuitive, but it's not a trivial property that holds for all types of estimators. For MLE, it's a guaranteed and exceptionally useful feature.

To get a clear conceptual overview of this property, please watch the following short segment from a video by zedstatistics.

2. A Formal Proof

Given your interest in mathematical rigor, let's now formally prove this property. The proof is quite elegant and relies directly on the definition of the MLE as the parameter value that maximizes the likelihood function. We'll focus on the case where the transformation function $g$ is one-to-one, as this is the clearest and most common scenario.

Let's summarize the logic of the proof you just read:

1. Setup: Let $\hat{\theta }$ be the unique MLE of $\theta$, meaning it maximizes the likelihood function $L(\theta )$. Let $\tau =g(\theta )$ be a one-to-one reparameterization, which implies an inverse function $\theta =g^{-1}(\tau )$ exists.

2. Reparameterize the Likelihood: The likelihood can be expressed in terms of the new parameter $\tau$ as $L^{\ast }(\tau )=L(g^{-1}(\tau ))$.

3. The Maximizer: Let $\hat{\tau }$ be the MLE for $\tau$, meaning it maximizes $L^{\ast }(\tau )$. Our goal is to show that $\hat{\tau }=g(\hat{\theta })$.

4. Proof by Contradiction: Assume $\hat{\tau }\ne g(\hat{\theta })$.

   • This implies that $g^{-1}(\hat{\tau })\ne \hat{\theta }$. Let's call ${\theta }_0=g^{-1}(\hat{\tau })$.
   • Since $\hat{\theta }$ is the unique maximizer of $L(\theta )$, it must be that $L(\hat{\theta })>L({\theta }_0)$.
   • Now, let's substitute the functions back in: $L(\hat{\theta })>L(g^{-1}(\hat{\tau }))$.
   • Using the definition of our reparameterized likelihood, this means $L(g^{-1}(g(\hat{\theta })))>L^{\ast }(\hat{\tau })$.
   • This simplifies to $L^{\ast }(g(\hat{\theta }))>L^{\ast }(\hat{\tau })$.

5. Conclusion: The last inequality, $L^{\ast }(g(\hat{\theta }))>L^{\ast }(\hat{\tau })$, states that the likelihood evaluated at $g(\hat{\theta })$ is greater than the likelihood evaluated at $\hat{\tau }$. This contradicts our initial statement that $\hat{\tau }$ is the value that maximizes the function $L^{\ast }(\tau )$. Therefore, our assumption was wrong, and it must be that $\hat{\tau }=g(\hat{\theta })$.

While this proof is for a one-to-one function, it's worth noting that the property holds even when the function $g$ is not one-to-one.

3. Invariance in Action: From Probability to Log-Odds

Let's see how this property saves us work in a more complex scenario. Consider data from a Bernoulli process (e.g., coin flips), where we estimate the probability of success, $\theta$. The MLE for $\theta$ is the sample mean: ${\hat{\theta }}_{MLE}=\bar{x}$.

In many statistical models, particularly logistic regression, it is more convenient to work with log-odds instead of probability. The odds are defined as $\frac{\theta }{1-\theta }$, and the log-odds are $\psi =\log \left(\frac{\theta }{1-\theta }\right)$. This transformation, $g(\theta )=\log \left(\frac{\theta }{1-\theta }\right)$, maps the probability from $(0,1)$ to the entire real line $(-\infty ,\infty )$, which has desirable properties for linear modeling.

Without the invariance property, you would have to:

1. Express $\theta$ in terms of $\psi$.
2. Substitute this into the Bernoulli likelihood function to get $L^{\ast }(\psi )$.
3. Differentiate $\log L^{\ast }(\psi )$ with respect to $\psi$.
4. Set the derivative to zero and solve for $\psi$.

With the invariance property, the solution is immediate. The MLE for the log-odds is:
${\hat{\psi }}_{MLE}=g({\hat{\theta }}_{MLE})=\log \left(\frac{{\hat{\theta }}_{MLE}}{1-{\hat{\theta }}_{MLE}}\right)=\log \left(\frac{\bar{x}}{1-\bar{x}}\right)$

The following video segment walks through this exact example and also introduces an important related concept: how to find the variance of this new, transformed estimator using the Delta Method. This connects beautifully to our next module, where we will rigorously evaluate the performance of estimators.

The Delta Method provides a way to perform a first-order Taylor approximation to find the variance of a transformed random variable. For a transformed estimator $\hat{\tau }=g(\hat{\theta })$, its variance is approximately:
$\text{Var}(\hat{\tau })\approx {\left[g^{\prime }(\theta )\right]}^2\text{Var}(\hat{\theta })$
This is a very useful technique in statistical inference, and we will see more of these concepts when we discuss the performance and properties of estimators in detail.

4. Hands-on Application

Let's apply the invariance principle to the coding exercise from our previous lesson. You wrote a script to find the MLEs for the parameters $\alpha$ and $\beta$ of a Beta distribution.

A key property of the Beta distribution is its mean, which is given by the function:
$\mu =g(\alpha ,\beta )=\frac{\alpha }{\alpha +\beta }$

Exercise:
Modify your Python script from the last lesson. After finding the MLEs $\hat{\alpha }$ and $\hat{\beta }$, use the invariance property to calculate the MLE for the mean of the Beta distribution, ${\hat{\mu }}_{MLE}$.

Add the following lines to the end of your script:

# ... (previous code for finding result.x)




# 7. Apply the invariance property to find the MLE for the mean
if result.success:
    estimated_alpha, estimated_beta = result.x
    



    # The function is g(alpha, beta) = alpha / (alpha + beta)
    estimated_mean_mle = estimated_alpha / (estimated_alpha + estimated_beta)
    



    # For comparison, calculate the actual sample mean
    sample_mean = np.mean(data)
    
    print(f"\nMLE for the mean (via invariance): {estimated_mean_mle:.4f}")
    print(f"Actual sample mean:                {sample_mean:.4f}")

You will notice that the MLE for the mean, calculated via the invariance property, is exactly equal to the sample mean of the data. This is not a coincidence! It turns out that for the Beta distribution, the Method of Moments estimator for the mean and the Maximum Likelihood Estimator for the mean are identical. The invariance property allows us to see this connection easily.

Conclusion

In this lesson, we explored the invariance property, a cornerstone of Maximum Likelihood Estimation. It provides a powerful and convenient method for finding the MLE of any function of a parameter without re-running the entire estimation procedure.

Key Takeaways:

• Invariance Property: If $\hat{\theta }$ is the MLE of $\theta$, the MLE of $g(\theta )$ is simply $g(\hat{\theta })$.
• Utility: This property greatly simplifies reparameterization, a common task in statistical modeling (e.g., converting variance to standard deviation, or probability to log-odds).
• Justification: The property is backed by a formal proof based on the definition of the MLE as a maximizer of the likelihood function.

Preview of the Next Module:

This lesson concludes our exploration of how to find estimators. We are now equipped with the Method of Moments, analytical MLE, and numerical MLE. The next module, "Optimality and Performance of Estimators," shifts our focus from finding estimators to evaluating them. We will ask critical questions: Is our estimator biased? What is its variance? Is there a theoretical "best" possible estimator? Answering these will lead us to two of the most important results in classical estimation theory: the Cramér-Rao Lower Bound (CRLB) and the Gauss-Markov theorem. These concepts will provide the rigorous benchmarks needed to assess the quality of the estimators you derive.