The Cramer-Rao Lower Bound: Conditions and Derivation

1. Intuition: Information, Likelihood Curvature, and Variance

Before diving into the mathematics, let's build some intuition. The precision with which we can estimate a parameter $\theta$ should logically depend on how much "information" about $\theta$ is contained in our data and our model.

Imagine two different likelihood functions derived from two different experiments, both aiming to estimate the same parameter $\theta$.

• Case 1: A sharply peaked likelihood function. This indicates that the data strongly supports a very narrow range of values for $\theta$. Moving away from the maximum likelihood estimate causes the likelihood to drop off rapidly. This implies our data is very informative, and we should be able to estimate $\theta$ with high precision (i.e., low variance).
• Case 2: A flat, broad likelihood function. Here, a wide range of $\theta$ values are almost equally plausible given the data. The likelihood changes slowly around its maximum. This implies our data is less informative, and our estimate will naturally have lower precision (i.e., higher variance).

The "sharpness" or "curvature" of the log-likelihood function at its peak is the key. A high curvature corresponds to high information and low variance. This concept is formalized by the Fisher Information.

To visualize this relationship, please watch the following video.

As the video explains, the curvature is captured by the second derivative of the log-likelihood function. The Fisher Information, $I(\theta )$, is defined as the negative of the expected value of this second derivative. A large Fisher Information value implies a sharply curved log-likelihood on average, meaning the data is rich in information about the parameter $\theta$.

2. Formalizing the CRLB: Regularity Conditions and Fisher Information

To derive the CRLB, we must first lay down the mathematical groundwork. The bound doesn't hold for all possible probability distributions; it requires some "regularity conditions" to be met. Given your interest in mathematical rigor, it's important to be aware of these.

In essence, the regularity conditions ensure that the model behaves well enough for the calculus in the proof to be valid:

1. The support of the distribution does not depend on the parameter $\theta$. For example, a Uniform distribution $U(0,\theta )$ violates this, as the range of possible data values depends on $\theta$. A Normal distribution $N(\theta ,1)$ satisfies this, as the data can take any real value regardless of $\theta$.
2. The operations of differentiation (with respect to $\theta$) and integration (with respect to $x$) can be interchanged. This is a mathematical "niceness" condition that holds for most distributions encountered in practice, including all members of the exponential family.

Fisher Information

With these conditions in place, we can formally define the key quantities. Please read the next section of the same document, which defines the score function and the Fisher Information.

Let's summarize the key definitions from that reading:

• Score function: The derivative of the log-likelihood function with respect to the parameter $\theta$. It tells us the gradient of the log-likelihood at any given $\theta$. $$S(X;\theta )=\frac{\partial }{\partial \theta }\log p(X;\theta )$$
• A Key Result (Lemma 1): The expected value of the score function is zero. $$E[S(X;\theta )]=0$$ The proof is elegant and relies on the regularity conditions allowing the interchange of differentiation and integration.
• Fisher Information: $I(\theta )$ can be expressed in two equivalent ways:
  1. As the variance of the score function: $$I(\theta )=E\left[{\left(\frac{\partial }{\partial \theta }\log p(X;\theta )\right)}^2\right]=\text{Var}(S(X;\theta ))$$ This follows directly from the score having zero mean ($\text{Var}(Z)=E[Z^2]-(E[Z]{)}^2$).
  2. As the negative expected second derivative of the log-likelihood: $$I(\theta )=-E\left[\frac{{\partial }^2}{\partial {\theta }^2}\log p(X;\theta )\right]$$ This second form directly captures the "expected curvature" we discussed intuitively and is often computationally simpler.

3. Derivation of the Cramér-Rao Lower Bound

We now have all the tools to derive the CRLB. The proof cleverly uses the Cauchy-Schwarz inequality to relate the variance of an estimator to the Fisher information.

For two random variables $U$ and $V$, the Cauchy-Schwarz inequality states:
$(\text{Cov}(U,V){)}^2\le \text{Var}(U)\text{Var}(V)$

We will apply this to our estimator $\hat{\theta }(X)$ and the score function $S(X;\theta )$.

Let's walk through the logic of the proof for an unbiased estimator $\hat{\theta }$ (i.e., $E[\hat{\theta }]=\theta$).

1. Set up the Covariance: We examine the covariance between the estimator $\hat{\theta }$ and the score $S$. Since $E[S]=0$, the covariance simplifies:
   $\text{Cov}(\hat{\theta },S)=E[\hat{\theta }S]-E[\hat{\theta }]E[S]=E[\hat{\theta }S]$

2. Evaluate the Expectation: We expand $E[\hat{\theta }S]$:

   $$E[\hat{\theta }S]=\int \hat{\theta }(x)\left(\frac{\partial }{\partial \theta }\log p(x;\theta )\right)p(x;\theta )dx$$

   Using the identity $\frac{\partial }{\partial \theta }\log p=\frac{1}{p}\frac{\partial p}{\partial \theta }$, this becomes:

   $$E[\hat{\theta }S]=\int \hat{\theta }(x)\frac{\partial p(x;\theta )}{\partial \theta }dx$$

   Using the regularity conditions, we swap the integral and derivative:

   $$E[\hat{\theta }S]=\frac{\partial }{\partial \theta }\int \hat{\theta }(x)p(x;\theta )dx=\frac{\partial }{\partial \theta }E[\hat{\theta }]$$

3. Use the Unbiased Property: Since $\hat{\theta }$ is unbiased, $E[\hat{\theta }]=\theta$. Therefore:

   $$\text{Cov}(\hat{\theta },S)=\frac{\partial }{\partial \theta }(\theta )=1$$

   This is a remarkable result: the covariance between any unbiased estimator and the score function is always 1.

4. Apply Cauchy-Schwarz: Now we apply the inequality:

   $$(\text{Cov}(\hat{\theta },S){)}^2\le \text{Var}(\hat{\theta })\text{Var}(S)$$

   Substituting our findings ($\text{Cov}(\hat{\theta },S)=1$ and $\text{Var}(S)=I(\theta )$):

   $$1^2\le \text{Var}(\hat{\theta })\cdot I(\theta )$$

5. The Final Result: Rearranging the terms gives the celebrated Cramér-Rao Lower Bound:

   $$\text{Var}(\hat{\theta })\ge \frac{1}{I(\theta )}$$

This inequality states that the variance of any unbiased estimator $\hat{\theta }$ must be greater than or equal to the reciprocal of the Fisher Information. No unbiased estimator can do better than this bound.

4. CRLB in Action: An Example

Let's make this concrete by seeing how it applies to a familiar problem. The following video walks through the entire process for estimating the mean of a Normal distribution.

The video shows the following key results for $n$ i.i.d. samples from $N(\mu ,{\sigma }^2)$:

• The Fisher Information for $\mu$ is $I(\mu )=\frac{n}{{\sigma }^2}$.
• The CRLB is therefore $\text{CRLB}=\frac{1}{I(\mu )}=\frac{{\sigma }^2}{n}$.

We know from basic statistics that if we use the sample mean $\hat{\mu }=\bar{X}$ as our estimator, its variance is $\text{Var}(\bar{X})=\frac{{\sigma }^2}{n}$.

Since $\text{Var}(\bar{X})=\text{CRLB}$, the sample mean achieves the lower bound. This proves that the sample mean is an efficient estimator—it is the Minimum Variance Unbiased Estimator (MVU) for the mean of a Normal distribution. We cannot find a better unbiased estimator.

Conclusion

In this lesson, you have delved into one of the most important results in classical estimation theory. You learned not only what the Cramér-Rao Lower Bound is, but also the formal conditions it requires and the mathematical steps to derive it.

Key Takeaways:

• The Cramér-Rao Lower Bound (CRLB) provides a theoretical minimum on the variance of any unbiased estimator.
• The bound is the reciprocal of the Fisher Information, $I(\theta )$, which quantifies the information about $\theta$ contained in the data.
• Fisher Information can be calculated either as the variance of the score function or as the negative expected curvature of the log-likelihood function.
• The derivation of the CRLB relies on the Cauchy-Schwarz inequality applied to the estimator and the score function.
• An unbiased estimator that achieves the CRLB is called efficient and is a Minimum Variance Unbiased Estimator (MVU).

Self-Test

To solidify your understanding, try to derive the CRLB for a different distribution. Your strong math background should make this an engaging exercise.

Problem: Consider $n$ i.i.d. samples $X_1,\dots ,X_n$ from a Poisson distribution with parameter $\lambda$. The PMF for a single observation is $p(x;\lambda )=\frac{e^{-\lambda }{\lambda }^x}{x!}$.

1. Write down the log-likelihood function for the $n$ samples.
2. Calculate the Fisher Information $I(\lambda )$. (Hint: The second derivative method is often easier here).
3. State the CRLB for any unbiased estimator of $\lambda$.
4. The sample mean $\hat{\lambda }=\bar{X}$ is an unbiased estimator for $\lambda$. Knowing that $\text{Var}(X_i)=\lambda$, is $\bar{X}$ an efficient estimator for $\lambda$?

Preview of the Next Lesson:

We've now established a powerful benchmark for estimator variance. In the next lesson, we will explore the significance of this bound further and introduce the Gauss-Markov theorem. This theorem provides a different kind of optimality guarantee, specifically for estimators in the context of linear models, and proves that the Ordinary Least Squares (OLS) estimator is the "Best Linear Unbiased Estimator" (BLUE) under certain conditions.