Likelihood Ratio Tests for Simple Hypotheses

1. The Principle of the Likelihood Ratio

Let's begin with a simple scenario. We have a set of observations $\mathbf{x}=(x_1,\dots ,x_n)$, and we want to test between two simple hypotheses:

• Null Hypothesis: $H_0:\theta ={\theta }_0$
• Alternative Hypothesis: $H_1:\theta ={\theta }_1$

The likelihood function, which you'll recall from our discussions on Maximum Likelihood Estimation, tells us the probability (or probability density) of observing our data $\mathbf{x}$ given a particular value of the parameter $\theta$. We can write the likelihood under each hypothesis:

• Likelihood under $H_0$: $L({\theta }_0|\mathbf{x})=p(\mathbf{x}|{\theta }_0)$
• Likelihood under $H_1$: $L({\theta }_1|\mathbf{x})=p(\mathbf{x}|{\theta }_1)$

A natural way to decide between these two hypotheses is to compare their likelihoods. This is precisely what the likelihood ratio does:

The decision rule follows logically:

• If $\Lambda (\mathbf{x})$ is very large, the data are much more likely under $H_1$ than $H_0$. This provides strong evidence against the null hypothesis, so we should reject $H_0$.
• If $\Lambda (\mathbf{x})$ is small (close to 0), the data are much more likely under $H_0$. This supports the null hypothesis, so we fail to reject $H_0$.

This leads to the formal Likelihood Ratio Test:

Reject $H_0$ if $\Lambda (\mathbf{x})>k$

where $k$ is a critical value or threshold that we must choose. But how do we choose $k$ in a non-arbitrary way? And can we prove that this is a good decision rule? This brings us to one of the most celebrated results in hypothesis testing.

2. The Neyman-Pearson Lemma: Deriving the Optimal Test

Instead of just picking a threshold $k$, let's frame the problem as an optimization, a perspective that should be familiar from your engineering background. We have two types of errors:

• Type I Error (False Alarm/False Positive): Rejecting $H_0$ when it is true. The probability is $\alpha =P(\text{reject }{H}_0|H_0\text{ is true})$.
• Type II Error (Miss/False Negative): Failing to reject $H_0$ when it is false. The probability is $\beta =P(\text{fail to reject }{H}_0|H_1\text{ is true})$.

The power of a test is the probability of correctly rejecting a false null hypothesis: $1-\beta$.

The Neyman-Pearson approach is to fix the probability of a Type I error at a certain level $\alpha$ (e.g., 5% or 1%) and find the test that maximizes the power (i.e., minimizes the Type II error $\beta$). This gives us the most powerful test for a given significance level $\alpha$.

The following video provides an elegant derivation of this optimal test using Lagrange multipliers, a technique you've likely used in mechanics and optimization.

As the video showed, maximizing the detection probability $P_D$ (power) subject to a fixed false alarm probability $P_{FA}$ ($\alpha$) leads to the decision rule: choose to reject $H_0$ in the region $R$ where $p_1(\mathbf{x})-\gamma p_0(\mathbf{x})>0$. Rearranging this gives:

This is exactly the likelihood ratio test. The Lagrange multiplier $\gamma$ becomes our decision threshold. We choose $\gamma$ such that the resulting test has the desired Type I error rate, $\alpha$.

This result is formally stated in the Neyman-Pearson Lemma. For a deep dive into the formal statement and its proof, the following document is an excellent resource.

3. Practical Derivation of the Test Statistic

The abstract form $\Lambda (\mathbf{x})>k$ is the principle, but in practice, we almost never compute the full likelihood ratio. Instead, we simplify the expression algebraically to get an equivalent test using a more convenient test statistic. Since the logarithm is a strictly increasing function, we can work with the log-likelihood ratio without changing the outcome of the test:

Reject $H_0$ if $\ln (\Lambda (\mathbf{x}))>\ln (k)$

Let's derive the test statistic for a common and important case.

Example: Mean of a Normal Distribution (Known Variance)

Suppose we have $n$ independent observations $X_1,\dots ,X_n$ from a normal distribution with known variance ${\sigma }^2$ and unknown mean $\mu$. We want to test:

• $H_0:\mu ={\mu }_0$
• $H_1:\mu ={\mu }_1$ (let's assume ${\mu }_1>{\mu }_0$)

1. Write the Likelihoods:
The PDF for a single observation $x_i$ is $f(x_i|\mu )=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)$.
Since the observations are independent, the joint likelihood is the product:
$L(\mu |\mathbf{x})={\prod }_{i=1}^n\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}\right)={\left(\frac{1}{2\pi {\sigma }^2}\right)}^{n/2}\exp \left(-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2\right)$

2. Form the Log-Likelihood Ratio:
$\ln (\Lambda (\mathbf{x}))=\ln \left(\frac{L({\mu }_1|\mathbf{x})}{L({\mu }_0|\mathbf{x})}\right)=\ln (L({\mu }_1|\mathbf{x}))-\ln (L({\mu }_0|\mathbf{x}))$
The log-likelihood is:
$\ln (L(\mu |\mathbf{x}))=-\frac{n}{2}\ln (2\pi {\sigma }^2)-\frac{1}{2{\sigma }^2}{\sum }_{i=1}^n(x_i-\mu {)}^2$
So the log-likelihood ratio becomes:
$\ln (\Lambda (\mathbf{x}))=\left(-\frac{1}{2{\sigma }^2}\sum (x_i-{\mu }_1{)}^2\right)-\left(-\frac{1}{2{\sigma }^2}\sum (x_i-{\mu }_0{)}^2\right)$
$=\frac{1}{2{\sigma }^2}\left[\sum (x_i-{\mu }_0{)}^2-\sum (x_i-{\mu }_1{)}^2\right]$

3. Simplify the Expression:
Let's expand the squared terms:
$\sum (x_i^2-2x_i{\mu }_0+{\mu }_0^2)-\sum (x_i^2-2x_i{\mu }_1+{\mu }_1^2)$
$=(-2{\mu }_0\sum x_i+n{\mu }_0^2)-(-2{\mu }_1\sum x_i+n{\mu }_1^2)$
$=2({\mu }_1-{\mu }_0)\sum x_i+n({\mu }_0^2-{\mu }_1^2)$
$=2({\mu }_1-{\mu }_0)\sum x_i+n({\mu }_0-{\mu }_1)({\mu }_0+{\mu }_1)$

4. Formulate the Test:
The LRT is to reject $H_0$ if $\ln (\Lambda (\mathbf{x}))>\ln (k)$. Substituting our simplified expression:
$\frac{1}{2{\sigma }^2}\left[2({\mu }_1-{\mu }_0)\sum x_i+n({\mu }_0^2-{\mu }_1^2)\right]>\ln (k)$
Now, we just isolate $\sum x_i$, or even better, the sample mean $\bar{x}=\frac{1}{n}\sum x_i$. Since we assumed ${\mu }_1>{\mu }_0$, the term $({\mu }_1-{\mu }_0)$ is positive, so the inequality direction is preserved.
$2({\mu }_1-{\mu }_0)n\bar{x}>2{\sigma }^2\ln (k)-n({\mu }_0^2-{\mu }_1^2)$
$\bar{x}>\frac{2{\sigma }^2\ln (k)-n({\mu }_0^2-{\mu }_1^2)}{2n({\mu }_1-{\mu }_0)}$
The entire right-hand side is just a constant that depends on ${\mu }_0,{\mu }_1,{\sigma }^2,n,$ and $k$. We can call this new threshold $k^{\prime }$.
$\bar{x}>k^{\prime }$
This is a remarkable result! The "most powerful test" boils down to a simple rule: reject the null hypothesis if the sample mean is greater than some critical value. This is far more intuitive and computationally simpler than calculating the full likelihood ratio. The threshold $k^{\prime }$ is chosen to give the desired significance level $\alpha$.

To see this derivation and another one for Bernoulli trials laid out, please review the following resource.

Conclusion

In this lesson, we bridged the gap between formulating hypotheses and making a decision. You derived the Likelihood Ratio Test, one of the cornerstones of statistical inference, and understood its foundational importance.

Key Takeaways:

• Likelihood Ratio ($\Lambda$): The ratio of the data's likelihood under the alternative hypothesis to its likelihood under the null hypothesis. It quantifies which hypothesis is better supported by the data.
• Likelihood Ratio Test (LRT): A decision rule that rejects $H_0$ if the likelihood ratio $\Lambda$ exceeds a certain threshold $k$.
• Neyman-Pearson Lemma: This fundamental theorem proves that for simple hypotheses, the LRT is the most powerful test. It yields the highest possible probability of detecting a true effect for a given acceptable rate of false alarms ($\alpha$).
• Practical Simplification: The abstract LRT often simplifies algebraically into an equivalent test on a simple, intuitive statistic like the sample mean ($\bar{x}$) or the number of successes.

Preview of the Next Lesson:

We have established a method for creating the best test for a given significance level $\alpha$. But what is the relationship between $\alpha$, the Type II error $\beta$, and the sample size $n$? In our next lesson, we will explore the mathematical trade-off between Type-I and Type-II errors and the concept of statistical power. This will allow us to not only perform tests but also to evaluate their performance and design effective experiments.